Oct 30 00:41:54.243581 ip-10-0-130-169 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:41:54.649810 ip-10-0-130-169 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:54.649810 ip-10-0-130-169 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:41:54.649810 ip-10-0-130-169 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:54.649810 ip-10-0-130-169 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:54.657079 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.656980    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.657079 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657074    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.657079 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657080    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.657079 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657084    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657089    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657094    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657098    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657102    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657107    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657111    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657115    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657122    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657129    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657134    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657138    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657142    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657146    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657150    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657153    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657157    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657162    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657166    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.657268 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657170    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657183    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657188    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657192    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657196    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657200    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657204    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657208    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657213    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657216    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657220    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657224    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657228    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657233    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657237    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657242    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657246    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657251    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657269    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.657733 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657274    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657279    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657283    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657287    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657291    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657296    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657301    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657305    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657310    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657315    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657319    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657323    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657327    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657331    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657335    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657341    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657345    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657349    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657353    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.658222 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657357    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657361    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657366    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657370    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657375    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657379    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657383    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657387    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657392    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657397    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657402    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657406    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657410    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657414    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657420    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657425    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657429    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657433    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657437    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657442    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.658687 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657446    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657450    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657454    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657461    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657465    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657469    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657473    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657477    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657481    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657485    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657489    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657493    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657497    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657501    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657505    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657509    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657514    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657518    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657522    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.659162 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657526    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657531    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657537    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657542    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657546    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657550    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.657555    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658632    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658641    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658646    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658650    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658655    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658659    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658664    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658668    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658672    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658676    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658682    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658686    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.659638 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658691    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658695    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658700    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658704    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658708    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658712    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658716    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658720    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658724    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658728    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658731    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658735    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658740    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658744    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658748    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658752    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658757    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658761    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658765    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658769    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.660101 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658773    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658777    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658781    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658785    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658788    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658793    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658797    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658801    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658805    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658809    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658813    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658817    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658822    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658828    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658834    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658838    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658843    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658847    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658852    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.660598 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658856    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658860    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658864    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658868    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658872    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658876    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658880    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658885    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658889    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658893    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658897    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658901    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658905    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658910    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658914    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658918    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658922    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658926    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658930    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658933    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.661053 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658938    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658942    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658946    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658949    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658953    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658958    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658962    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658966    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658970    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658974    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658978    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658982    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658986    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658990    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658994    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.658999    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659003    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659007    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659012    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659015    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.661562 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659019    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659023    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659027    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659031    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659035    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659039    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659043    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659047    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659051    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659055    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659059    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659063    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659070    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659075    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659080    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659799    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659808    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659812    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659817    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659821    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.662038 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659825    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659829    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659833    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659837    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659841    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659846    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659850    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659854    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659858    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659862    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659866    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659871    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659875    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659879    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659884    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659889    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659893    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659898    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659902    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659906    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.662534 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659909    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659913    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659918    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659922    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659926    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659930    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659934    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659938    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659942    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659946    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659951    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659956    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659959    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659963    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659967    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659971    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659975    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659980    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659984    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659988    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.663011 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659992    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.659996    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660000    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660004    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660008    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660012    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660016    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660020    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660024    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660029    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660033    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660037    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660041    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660045    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660050    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660054    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660058    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660062    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660066    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.663526 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660071    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660076    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660080    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660084    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660089    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660093    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660097    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660101    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660105    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660111    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660116    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660121    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660126    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660130    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660135    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660138    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660142    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660146    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660150    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660154    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.664008 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660159    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660163    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660167    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660171    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660175    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660179    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660183    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660187    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660192    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660196    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660200    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660205    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660211    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660218    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660223    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660227    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660232    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660237    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660241    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.664494 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660245    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660249    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.660268    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661226    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661239    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661249    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661268    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661275    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661280    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661290    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661297    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661302    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661306    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661313    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661318    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661323    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661327    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661332    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661337    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661341    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661346    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661352    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661356    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:41:54.665045 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661361    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661366    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661371    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661378    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661383    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661388    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661393    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661398    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661403    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661407    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661413    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661418    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661424    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661429    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661433    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661438    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661443    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661447    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661455    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661460    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661464    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661469    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661474    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661482    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:41:54.665657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661486    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661491    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661496    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661501    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661506    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661510    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661515    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661519    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661525    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661530    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661537    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661541    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661546    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661550    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661556    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661560    2579 flags.go:64] FLAG: --help="false"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661564    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-169.ec2.internal"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661571    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661576    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661581    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661586    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661591    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661595    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:41:54.666296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661600    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661604    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661609    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661614    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661619    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661623    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661628    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661633    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661664    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661670    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661676    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661681    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661686    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661690    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661699    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661704    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661709    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661714    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661719    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661724    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661728    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661733    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661740    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661744    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661751    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:41:54.666846 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661755    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661760    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661765    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661770    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661775    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661779    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661784    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661796    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661800    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661805    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661811    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661815    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661819    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661824    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661829    2579 flags.go:64] FLAG: --port="10250"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661834    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661839    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0377050885885c137"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661844    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661849    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661854    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661859    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661866    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661871    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661875    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661882    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661888    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:41:54.667478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661892    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661897    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661902    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661906    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661911    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661916    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661920    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661925    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661930    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661935    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661940    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661945    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661949    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661954    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661958    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661962    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661967    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661972    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661977    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661984    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661989    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.661993    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662000    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662004    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662007    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:41:54.668094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662012    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662017    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662021    2579 flags.go:64] FLAG: --v="2"
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662028    2579 flags.go:64] FLAG: --version="false"
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662033    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662043    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662048    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662243    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662249    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662267    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662272    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662276    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662281    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662285    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662289    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662294    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662298    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662304    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662308    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662312    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662316    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.668706 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662321    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662325    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662329    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662333    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662337    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662341    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662345    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662349    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662353    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662357    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662361    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662365    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662369    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662373    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662378    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662382    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662390    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662395    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662401    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662405    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.669201 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662410    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662414    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662418    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662422    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662426    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662430    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662435    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662439    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662442    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662447    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662451    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662456    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662460    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662464    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662468    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662472    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662478    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662484    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662488    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.669743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662493    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662497    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662501    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662508    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662513    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662518    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662522    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662526    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662530    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662536    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662540    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662546    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662550    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662554    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662559    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662563    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662568    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662572    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662575    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662580    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.670400 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662584    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662588    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662592    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662603    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662607    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662611    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662615    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662618    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662623    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662628    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662632    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662636    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662640    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662644    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662648    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662652    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662657    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662661    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662665    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662669    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.670894 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662673    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662679    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662684    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662690    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662695    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662699    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662703    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662707    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662711    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662716    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662720    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662724    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662728    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.662765    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662823    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662830    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.671403 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662835    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662839    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662843    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662848    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662851    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662856    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662860    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662864    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662869    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662873    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662877    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662881    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662886    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662889    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662894    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662898    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662902    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662906    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662912    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662918    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.671799 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662923    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662927    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662931    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662935    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662939    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662943    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662947    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662952    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662956    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662960    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662965    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662969    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662973    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662978    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662982    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662986    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662990    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662995    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.662999    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663003    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.672304 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663007    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663011    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663016    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663021    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663025    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663029    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663033    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663037    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663041    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663045    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663052    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663058    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663062    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663066    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663070    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663074    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663079    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663083    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663087    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.672795 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663091    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663095    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663099    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663103    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663108    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663112    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663117    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663122    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663126    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663131    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663135    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663141    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663146    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663150    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663154    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663158    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663162    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663168    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663174    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.673248 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663178    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663183    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663187    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663191    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663197    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663203    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663207    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663211    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663215    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663219    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663223    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663228    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663232    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663236    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663240    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663244    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663249    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663266    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663270    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663274    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.673715 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663278    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663283    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663289    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663293    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663299    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.663303    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.671828    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.671846    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.671861    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.671866    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671939    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671945    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671948    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671953    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671956    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671959    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671962    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671965    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671967    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671970    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.674207 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671973    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671976    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671979    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671981    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671984    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671987    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671990    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671992    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671995    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.671998    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672001    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672003    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672006    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672009    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672013    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672016    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672019    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672021    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672024    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.674709 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672028    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672030    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672033    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672036    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672038    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672041    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672044    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672046    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672049    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672051    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672054    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672057    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672059    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672062    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672064    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672067    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672071    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672074    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672077    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.675153 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672080    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672083    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672085    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672088    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672091    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672093    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672096    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672099    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672102    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672104    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672107    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672110    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672113    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672118    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672122    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672125    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672128    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672131    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672134    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.675662 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672137    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672140    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672142    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672145    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672148    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672151    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672153    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672156    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672159    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672162    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672164    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672167    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672170    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672172    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672175    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672177    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672180    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672182    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672185    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.676119 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672187    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672190    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672193    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672202    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672205    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672208    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672211    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672213    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672216    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672219    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672222    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672225    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672228    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672231    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672233    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672236    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672239    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672241    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672244    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.676592 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672246    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.672299    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672460    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672465    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672468    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672470    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672473    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672476    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672478    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672481    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672483    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672486    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672489    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672491    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672494    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672497    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.677047 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672500    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672503    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672506    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672509    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672512    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672514    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672517    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672519    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672522    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672525    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672527    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672530    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672532    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672535    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672538    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672540    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672543    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672545    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672548    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672550    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.677444 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672553    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672555    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672558    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672560    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672563    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672566    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672568    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672571    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672574    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672576    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672579    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672582    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672584    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672588    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672593    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672597    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672600    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672603    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672606    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.677926 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672608    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672611    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672614    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672616    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672619    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672622    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672625    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672627    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672630    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672632    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672635    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672638    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672640    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672643    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672645    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672648    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672651    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672653    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672656    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.678397 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672659    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672663    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672666    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672670    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672673    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672676    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672679    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672681    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672684    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672687    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672690    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672692    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672695    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672698    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672700    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672703    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672705    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672708    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672710    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672713    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.678866 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672716    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672718    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672720    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672723    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672726    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672729    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672731    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672734    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672736    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672739    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672742    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672744    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672747    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672749    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.672772    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:54.679366 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672803    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672808    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672811    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672814    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672817    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672819    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672822    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672825    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672828    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672832    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672834    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672837    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672842    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672845    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672848    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672851    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672853    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672856    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672859    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672861    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:54.679743 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672864    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672867    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672869    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672872    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672875    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672878    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672881    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672884    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672887    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672889    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672892    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672894    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672898    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672902    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672905    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672909    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672912    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672915    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672918    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:54.680211 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672920    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672924    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672926    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672929    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672932    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672936    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672939    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672942    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672944    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672947    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672950    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672952    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672955    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672958    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672960    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672963    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672966    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672969    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672971    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:54.680669 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672976    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672980    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672983    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672986    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672989    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672991    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672994    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672996    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.672999    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673002    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673004    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673007    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673010    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673012    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673015    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673018    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673020    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673023    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673026    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673029    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:54.681115 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673032    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673035    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673037    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673040    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673043    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673045    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673048    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673051    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673055    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673058    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673061    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673063    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673066    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673068    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673071    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673074    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673076    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673079    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673081    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:54.681607 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673084    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673087    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673090    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673092    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673095    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673097    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673100    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673102    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:54.673105    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.673160    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.677645    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.678734    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.678831    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.679534    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:41:54.682204 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.680915    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:41:54.684761 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.684735    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:41:54.698209 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.698188    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:41:54.702232 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.702213    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 00:41:54.703113 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.703084    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:41:54.707421 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.707400    2579 fs.go:135] Filesystem UUIDs: map[2adee509-3055-4353-8f3d-e2b44b78f014:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c9e86c4c-d3e5-43b6-9f5a-19788d739546:/dev/nvme0n1p4]
Oct 30 00:41:54.707505 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.707421    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:41:54.712810 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.712690    2579 manager.go:217] Machine: {Timestamp:2026-10-30 00:41:54.711187521 +0000 UTC m=+0.383893448 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098826 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0d4b17ceaff9a33d0fe954c0bde8 SystemUUID:ec2e0d4b-17ce-aff9-a33d-0fe954c0bde8 BootID:25ef4616-8dbf-466e-9a9b-8e601af465d3 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:79:1c:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:79:1c:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:d9:ad:ec:a7:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:41:54.713591 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.713579    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:41:54.713687 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.713674    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:41:54.716465 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.716434    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:41:54.716614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.716466    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-169.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:41:54.716668 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.716616    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:41:54.716668 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.716625    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:41:54.716668 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.716658    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:41:54.716744 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.716677    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:41:54.717936 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.717924    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:41:54.722727 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.722711    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:41:54.722850 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.722840    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:41:54.723670 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.723660    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:41:54.723705 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.723674    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:41:54.723705 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.723684    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:41:54.723705 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.723693    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:41:54.723705 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.723701    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:41:54.724223 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.724171    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:41:54.726804 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.726786    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:41:54.734544 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.734527    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:41:54.736418 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736400    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:41:54.736496 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736434    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:41:54.736605 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736594    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:41:54.736636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736611    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:41:54.736636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736617    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:41:54.736636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736623    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:41:54.736636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736628    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:41:54.736636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.736634    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:41:54.737355 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.737343    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:41:54.737393 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.737356    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:41:54.737393 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.737365    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:41:54.737393 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.737372    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:41:54.737393 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.737382    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:41:54.737393 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.737393    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:41:54.738202 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.738191    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:41:54.738238 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.738204    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:41:54.742477 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.742462    2579 server.go:1263] "Started kubelet"
Oct 30 00:41:54.742639 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.742584    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:41:54.742818 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.742704    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:41:54.742818 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.742799    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:41:54.743316 ip-10-0-130-169 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:41:54.744390 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.744372    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:41:54.744929 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.744912    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:41:54.748322 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.748295    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:41:54.751704 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.751681    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:41:54.751803 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.751682    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:41:54.751941 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.751925    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-169.ec2.internal\" not found"
Oct 30 00:41:54.752626 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752606    2579 factory.go:55] Registering systemd factory
Oct 30 00:41:54.752626 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752627    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:41:54.752776 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752655    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:41:54.752776 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752712    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:41:54.752776 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752729    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:41:54.752883 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752838    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:41:54.752883 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752848    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:41:54.752951 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752888    2579 factory.go:153] Registering CRI-O factory
Oct 30 00:41:54.752951 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752903    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:41:54.753019 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752956    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:41:54.753019 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752976    2579 factory.go:103] Registering Raw factory
Oct 30 00:41:54.753019 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.752990    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:41:54.754132 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.754109    2579 manager.go:319] Starting recovery of all containers
Oct 30 00:41:54.765337 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.765314    2579 manager.go:324] Recovery completed
Oct 30 00:41:54.768866 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.768795    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:41:54.768983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.768956    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-169.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:41:54.769492 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.769431    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:41:54.772443 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.768877    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-169.ec2.internal.18e327dedb802e72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-169.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-169.ec2.internal,},FirstTimestamp:2026-10-30 00:41:54.742423154 +0000 UTC m=+0.415129080,LastTimestamp:2026-10-30 00:41:54.742423154 +0000 UTC m=+0.415129080,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-169.ec2.internal,}"
Oct 30 00:41:54.773248 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.773235    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:54.775192 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.775170    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:54.775278 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.775211    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:54.775278 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.775226    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:54.775966 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.775947    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:41:54.775966 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.775962    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:41:54.776115 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.775983    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:41:54.776115 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.776005    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:54.776786 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.776770    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:54.776848 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.776796    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:54.776848 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.776807    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:54.778319 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.778306    2579 policy_none.go:50] "Start"
Oct 30 00:41:54.778319 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.778321    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:41:54.778415 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.778339    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:41:54.780330 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.780318    2579 policy_none.go:44] "Start"
Oct 30 00:41:54.785645 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.785506    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:41:54.785895 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.785864    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:41:54.786328 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.786307    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mqzr8"
Oct 30 00:41:54.793575 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.793555    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mqzr8"
Oct 30 00:41:54.808852 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.808630    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.813041    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.813076    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.813088    2579 server.go:87] "Starting device plugin registration server"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.813340    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.813350    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.813452    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.813534    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.813543    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.814191    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.814241    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-169.ec2.internal\" not found"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.817061    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.826193    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.827419    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.827453    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.827472    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:41:54.831830 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.827506    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 00:41:54.842755 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.842738    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:54.914379 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.914315    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:54.915375 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.915351    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:54.915502 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.915387    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:54.915502 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.915398    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:54.915502 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.915423    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-169.ec2.internal"
Oct 30 00:41:54.928116 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.928086    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"]
Oct 30 00:41:54.928236 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.928154    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:54.928966 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.928951    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:54.929027 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.928979    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:54.929027 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.928991    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:54.931293 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.931280    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:54.931459 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.931443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:54.931500 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.931478    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:54.932061 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.932042    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:54.932162 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.932045    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:54.932162 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.932075    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:54.932162 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.932089    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:54.932162 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.932090    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:54.932162 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.932104    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:54.934316 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.934299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:54.934422 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.934323    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:54.935126 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.935110    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:54.935203 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.935150    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:54.935203 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.935165    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:54.935633 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.935617    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-169.ec2.internal"
Oct 30 00:41:54.935719 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:54.935642    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-169.ec2.internal\": node \"ip-10-0-130-169.ec2.internal\" not found"
Oct 30 00:41:54.953165 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.953137    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:54.963618 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.963599    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:54.966343 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:54.966324    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.048812 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.048781    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:55.049406 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.049390    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:55.051472 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.051446    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:55.051580 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:55.051495    2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.051580 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.051511    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.053640 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.053622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.053707 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.053649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2831292250175cd4fa5c7c5f1d89a07-config\") pod \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" (UID: \"c2831292250175cd4fa5c7c5f1d89a07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.053707 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.053667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.073178 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.073156    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:55.073287 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:55.073197    2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.154761 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.154728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.154931 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.154767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.154931 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.154820    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.154931 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.154828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2831292250175cd4fa5c7c5f1d89a07-config\") pod \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" (UID: \"c2831292250175cd4fa5c7c5f1d89a07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.154931 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.154878    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:55.154931 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:55.154889    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2831292250175cd4fa5c7c5f1d89a07-config\") pod \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" (UID: \"c2831292250175cd4fa5c7c5f1d89a07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:57.519922 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.519883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:57.520985 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.520967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:41:57.842691 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.842612    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:41:57.843153 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.842798    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:41:57.888653 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.888623    2579 apiserver.go:52] "Watching apiserver"
Oct 30 00:41:57.916222 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.916193    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:41:57.936292 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.936252    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:41:57.936605 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.936582    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal","kube-system/global-pull-secret-syncer-nk4k6","kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal","openshift-dns/node-resolver-rxj86","openshift-multus/multus-tcqrb","openshift-cluster-node-tuning-operator/tuned-ztnp7","openshift-multus/network-metrics-daemon-2q7g6","openshift-network-diagnostics/network-check-target-77hhk","openshift-ovn-kubernetes/ovnkube-node-tntr4","kube-system/konnectivity-agent-skbfn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7","openshift-multus/multus-additional-cni-plugins-b6zxp","openshift-network-operator/iptables-alerter-6ns76","openshift-image-registry/node-ca-r6gln"]
Oct 30 00:41:57.939108 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.939091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:57.939192 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:57.939173    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:41:57.941081 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.941067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:57.943080 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.943066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:57.945085 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.945072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:57.947116 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.947097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:57.947191 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:57.947172    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:41:57.949167 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.949153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:41:57.949213 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:57.949201    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:41:57.951373 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.951359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:57.953445 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.953426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:57.955721 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.955701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:57.957316 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.957290    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:41:57.957392 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.957348    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:41:57.957750 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.957733    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hp8gq\""
Oct 30 00:41:57.958008 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.957993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:57.958562 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.958543    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:41:57.958657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.958595    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:41:57.960274 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.960255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:57.962570 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.962553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:57.963226 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.963206    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:41:57.963689 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.963657    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:36:56 +0000 UTC" deadline="2028-06-04 01:54:36.334961347 +0000 UTC"
Oct 30 00:41:57.963689 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.963687    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13993h12m38.37127604s"
Oct 30 00:41:57.995446 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.995426    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:57.998606 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:57.998591    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:58.018092 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.018074    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:41:58.023774 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.023746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-netns\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.023910 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.023791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7469affe-9d00-44b3-a833-4ca25b86da5c-konnectivity-ca\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:58.023910 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.023816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-host\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.023910 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.023856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.023910 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.023893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d027727-5c89-44a2-b1b8-20661fd13909-tmp-dir\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.024082 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.023918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9nl\" (UniqueName: \"kubernetes.io/projected/4254ef03-935f-47d4-948a-70855cb3afb8-kube-api-access-4l9nl\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.024082 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.023959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.024082 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.024230 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-sys-fs\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.024230 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-system-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.024230 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-socket-dir-parent\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.024230 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:58.024230 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-systemd-units\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.024471 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-multus-certs\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.024471 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:41:58.024471 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.024471 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-kubernetes\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.024471 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-tuned\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.024471 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovn-node-metrics-cert\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.024682 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-cni-binary-copy\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.024682 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-host\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.024682 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg87z\" (UniqueName: \"kubernetes.io/projected/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-kube-api-access-cg87z\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:58.024682 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.024682 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62hj\" (UniqueName: \"kubernetes.io/projected/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-kube-api-access-p62hj\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.024682 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-tmp\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-etc-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-config\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-env-overrides\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgzs\" (UniqueName: \"kubernetes.io/projected/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-kube-api-access-vdgzs\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-systemd\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/927717eb-ca5c-4f65-8a7a-bdc83054080c-iptables-alerter-script\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.024935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-run\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.024956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-var-lib-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7469affe-9d00-44b3-a833-4ca25b86da5c-agent-certs\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-cnibin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/927717eb-ca5c-4f65-8a7a-bdc83054080c-host-slash\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysconfig\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-slash\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-log-socket\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-script-lib\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d027727-5c89-44a2-b1b8-20661fd13909-hosts-file\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6b25\" (UniqueName: \"kubernetes.io/projected/927717eb-ca5c-4f65-8a7a-bdc83054080c-kube-api-access-l6b25\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-kubelet-config\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-netns\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-bin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-multus\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7j55\" (UniqueName: \"kubernetes.io/projected/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kube-api-access-p7j55\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.025583 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-lib-modules\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cnibin\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-sys\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-var-lib-kubelet\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-serviceca\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-system-cni-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-modprobe-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgrf\" (UniqueName: \"kubernetes.io/projected/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-kube-api-access-qrgrf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-kubelet\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.025983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.025985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-ovn\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8tx7\" (UniqueName: \"kubernetes.io/projected/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-kube-api-access-s8tx7\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-kubelet\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-socket-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7cp\" (UniqueName: \"kubernetes.io/projected/7d027727-5c89-44a2-b1b8-20661fd13909-kube-api-access-2r7cp\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-conf-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-node-log\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-device-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-dbus\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-multus-daemon-config\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-registration-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-systemd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-hostroot\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-etc-kubernetes\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.026436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-conf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.026945 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-bin\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.026945 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-netd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.026945 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.026554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-k8s-cni-cncf-io\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.058945 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.058919    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:41:58.059177 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.058987    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ppwp\""
Oct 30 00:41:58.059177 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059037    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:41:58.059177 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059049    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:41:58.059177 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059098    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:58.059177 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059097    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:58.059177 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059130    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:41:58.059402 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059265    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:41:58.059402 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059387    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfrd4\""
Oct 30 00:41:58.059719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059697    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:41:58.059719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059714    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7mtf\""
Oct 30 00:41:58.059841 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.059732    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:41:58.061800 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.061783    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnsxr\""
Oct 30 00:41:58.061903 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.061834    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cd6zv\""
Oct 30 00:41:58.072734 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.072712    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:41:58.073436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073417    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:41:58.073436 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073429    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073469    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073429    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073489    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073418    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073493    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073506    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tlfwv\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073515    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073419    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073556    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tgxx\""
Oct 30 00:41:58.073614 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073573    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:41:58.073962 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073670    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:41:58.073962 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073731    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7gc74"
Oct 30 00:41:58.073962 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.073745    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:41:58.076150 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.076127    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-stkll\""
Oct 30 00:41:58.076259 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.076243    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:41:58.127193 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-etc-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127193 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-config\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127193 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-env-overrides\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127193 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgzs\" (UniqueName: \"kubernetes.io/projected/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-kube-api-access-vdgzs\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-systemd\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127258    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-etc-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/927717eb-ca5c-4f65-8a7a-bdc83054080c-iptables-alerter-script\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127294    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-systemd\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-run\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-var-lib-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127402    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-var-lib-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127478 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7469affe-9d00-44b3-a833-4ca25b86da5c-agent-certs\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-cnibin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127523    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-run\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127581    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/927717eb-ca5c-4f65-8a7a-bdc83054080c-host-slash\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127642    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-cnibin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127682    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/927717eb-ca5c-4f65-8a7a-bdc83054080c-host-slash\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysconfig\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-slash\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-log-socket\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-script-lib\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d027727-5c89-44a2-b1b8-20661fd13909-hosts-file\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6b25\" (UniqueName: \"kubernetes.io/projected/927717eb-ca5c-4f65-8a7a-bdc83054080c-kube-api-access-l6b25\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-kubelet-config\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-netns\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127849    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-log-socket\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.127952 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-bin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127887    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d027727-5c89-44a2-b1b8-20661fd13909-hosts-file\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127904    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysconfig\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127934    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-multus\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-netns\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127971    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127984    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/927717eb-ca5c-4f65-8a7a-bdc83054080c-iptables-alerter-script\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128007    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-env-overrides\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-multus\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-slash\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-kubelet-config\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7j55\" (UniqueName: \"kubernetes.io/projected/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kube-api-access-p7j55\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-lib-modules\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128052    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.127987    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-bin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cnibin\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-sys\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128177    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-sys\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.128749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128206    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cnibin\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128225    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-config\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-var-lib-kubelet\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128293    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-lib-modules\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-serviceca\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128335    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128343    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-var-lib-kubelet\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-system-cni-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-modprobe-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgrf\" (UniqueName: \"kubernetes.io/projected/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-kube-api-access-qrgrf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-kubelet\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128434    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-system-cni-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-ovn\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128451    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-script-lib\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8tx7\" (UniqueName: \"kubernetes.io/projected/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-kube-api-access-s8tx7\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.129538 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128508    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-modprobe-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128515    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-kubelet\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-kubelet\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128546    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128553    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-kubelet\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-socket-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128549    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-ovn\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128641    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7cp\" (UniqueName: \"kubernetes.io/projected/7d027727-5c89-44a2-b1b8-20661fd13909-kube-api-access-2r7cp\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-conf-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-node-log\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-device-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-dbus\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128740    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-node-log\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128745    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-conf-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-multus-daemon-config\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.130189 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128789    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-device-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128807    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-dbus\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-registration-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-systemd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128851    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-hostroot\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128889    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-systemd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128901    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-registration-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-etc-kubernetes\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128927    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-etc-kubernetes\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-conf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128939    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-hostroot\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-bin\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-netd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.128992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-k8s-cni-cncf-io\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-netns\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129026    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-conf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.131051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7469affe-9d00-44b3-a833-4ca25b86da5c-konnectivity-ca\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-host\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129059    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-k8s-cni-cncf-io\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129071    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-bin\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129077    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-netns\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d027727-5c89-44a2-b1b8-20661fd13909-tmp-dir\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129136    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-netd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129122    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-host\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9nl\" (UniqueName: \"kubernetes.io/projected/4254ef03-935f-47d4-948a-70855cb3afb8-kube-api-access-4l9nl\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.129162    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.129242    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret podName:b3bb0053-40b7-49df-9490-192e78a9b720 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:58.629205301 +0000 UTC m=+2.137712183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret") pod "global-pull-secret-syncer-nk4k6" (UID: "b3bb0053-40b7-49df-9490-192e78a9b720") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-sys-fs\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129285    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-socket-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-system-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.131943 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-socket-dir-parent\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-systemd-units\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-multus-certs\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-system-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129529    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-multus-certs\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-kubernetes\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129561    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-serviceca\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-tuned\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovn-node-metrics-cert\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.129634    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129666    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-socket-dir-parent\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129682    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.129705    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs podName:ce14d2cc-a9e8-49af-be37-9b8e6a433c65 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:58.62966824 +0000 UTC m=+2.138175120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs") pod "network-metrics-daemon-2q7g6" (UID: "ce14d2cc-a9e8-49af-be37-9b8e6a433c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-cni-binary-copy\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129731    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-sys-fs\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d027727-5c89-44a2-b1b8-20661fd13909-tmp-dir\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129753    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-systemd-units\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-host\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg87z\" (UniqueName: \"kubernetes.io/projected/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-kube-api-access-cg87z\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129822    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62hj\" (UniqueName: \"kubernetes.io/projected/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-kube-api-access-p62hj\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129876    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-host\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-tmp\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.129920    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.130144    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-kubernetes\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.130222    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.130619    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-multus-daemon-config\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.131047    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7469affe-9d00-44b3-a833-4ca25b86da5c-konnectivity-ca\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.131416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-cni-binary-copy\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.131912    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-tuned\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.132253    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovn-node-metrics-cert\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.132983 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.132310    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7469affe-9d00-44b3-a833-4ca25b86da5c-agent-certs\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:58.133516 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.132716    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-tmp\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.148359 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.148323    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6b25\" (UniqueName: \"kubernetes.io/projected/927717eb-ca5c-4f65-8a7a-bdc83054080c-kube-api-access-l6b25\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.148587 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.148558    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgzs\" (UniqueName: \"kubernetes.io/projected/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-kube-api-access-vdgzs\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.148742 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.148647    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:41:58.148742 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.148672    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:41:58.148742 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.148689    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rsphz for pod openshift-network-diagnostics/network-check-target-77hhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:58.148884 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.148753    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz podName:f69dcc85-0b97-43eb-8674-0ec1415c8dfb nodeName:}" failed. No retries permitted until 2026-10-30 00:41:58.648734889 +0000 UTC m=+2.157241767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rsphz" (UniqueName: "kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz") pod "network-check-target-77hhk" (UID: "f69dcc85-0b97-43eb-8674-0ec1415c8dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:58.149690 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.149661    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7j55\" (UniqueName: \"kubernetes.io/projected/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kube-api-access-p7j55\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.149888 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.149864    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgrf\" (UniqueName: \"kubernetes.io/projected/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-kube-api-access-qrgrf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.150358 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.150333    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8tx7\" (UniqueName: \"kubernetes.io/projected/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-kube-api-access-s8tx7\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.150434 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.150412    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7cp\" (UniqueName: \"kubernetes.io/projected/7d027727-5c89-44a2-b1b8-20661fd13909-kube-api-access-2r7cp\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.151250 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.151233    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9nl\" (UniqueName: \"kubernetes.io/projected/4254ef03-935f-47d4-948a-70855cb3afb8-kube-api-access-4l9nl\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.151639 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.151615    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62hj\" (UniqueName: \"kubernetes.io/projected/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-kube-api-access-p62hj\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.151709 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.151685    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg87z\" (UniqueName: \"kubernetes.io/projected/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-kube-api-access-cg87z\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:58.188619 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.188581    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2831292250175cd4fa5c7c5f1d89a07.slice/crio-37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b WatchSource:0}: Error finding container 37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b: Status 404 returned error can't find the container with id 37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b
Oct 30 00:41:58.188992 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.188968    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf851886b8d1abdda99ba2c15c29f78.slice/crio-19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d WatchSource:0}: Error finding container 19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d: Status 404 returned error can't find the container with id 19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d
Oct 30 00:41:58.194421 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.194400    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:41:58.278211 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.278181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:41:58.285314 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.285287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tcqrb"
Oct 30 00:41:58.288604 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.288579    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d027727_5c89_44a2_b1b8_20661fd13909.slice/crio-bdbb2c8206606e39d673e1c3da53ada1679b600e07a2235916f856cfc32041b6 WatchSource:0}: Error finding container bdbb2c8206606e39d673e1c3da53ada1679b600e07a2235916f856cfc32041b6: Status 404 returned error can't find the container with id bdbb2c8206606e39d673e1c3da53ada1679b600e07a2235916f856cfc32041b6
Oct 30 00:41:58.294579 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.294555    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4254ef03_935f_47d4_948a_70855cb3afb8.slice/crio-64e2b2035554c6d65d1cb62e61ebe849dbc427fd3a978173782c3a3df331a3d1 WatchSource:0}: Error finding container 64e2b2035554c6d65d1cb62e61ebe849dbc427fd3a978173782c3a3df331a3d1: Status 404 returned error can't find the container with id 64e2b2035554c6d65d1cb62e61ebe849dbc427fd3a978173782c3a3df331a3d1
Oct 30 00:41:58.306777 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.306750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:41:58.315913 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.315883    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ee8ed6_aae3_42a9_adfe_3ecbfd348141.slice/crio-9c7c2708d7db6bd2afb1525e76a4c34188d906fe37a9b22a3409f4b3a35cb186 WatchSource:0}: Error finding container 9c7c2708d7db6bd2afb1525e76a4c34188d906fe37a9b22a3409f4b3a35cb186: Status 404 returned error can't find the container with id 9c7c2708d7db6bd2afb1525e76a4c34188d906fe37a9b22a3409f4b3a35cb186
Oct 30 00:41:58.328998 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.328976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:41:58.331160 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.331137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:41:58.339531 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.339508    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7469affe_9d00_44b3_a833_4ca25b86da5c.slice/crio-615b46a4c54925eacc7cbf15bec37c7ad0c000436b828c0301bbab676933ecca WatchSource:0}: Error finding container 615b46a4c54925eacc7cbf15bec37c7ad0c000436b828c0301bbab676933ecca: Status 404 returned error can't find the container with id 615b46a4c54925eacc7cbf15bec37c7ad0c000436b828c0301bbab676933ecca
Oct 30 00:41:58.340059 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.340040    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085c11e9_3ae8_4a1d_abe7_edc7dcad9804.slice/crio-303442b5c031dfa67e64696ad482f1f3466163bbf51b19d452685f0a7663048e WatchSource:0}: Error finding container 303442b5c031dfa67e64696ad482f1f3466163bbf51b19d452685f0a7663048e: Status 404 returned error can't find the container with id 303442b5c031dfa67e64696ad482f1f3466163bbf51b19d452685f0a7663048e
Oct 30 00:41:58.348023 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.348005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:41:58.357605 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.357583    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e082e1_ce44_4deb_8036_89d2ae8dcc46.slice/crio-501238e2345bfe4f9ebfa0008d8f4b12e0993b7905f8f4bad3f7f3393b72ccc7 WatchSource:0}: Error finding container 501238e2345bfe4f9ebfa0008d8f4b12e0993b7905f8f4bad3f7f3393b72ccc7: Status 404 returned error can't find the container with id 501238e2345bfe4f9ebfa0008d8f4b12e0993b7905f8f4bad3f7f3393b72ccc7
Oct 30 00:41:58.362355 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.362338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:41:58.368109 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.368088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:41:58.371318 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.371297    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d8e7da_34ee_466e_bf3d_48a8a56b722f.slice/crio-96fae5a4031880a78bac9a755c06367fa3a998e276d66611b25f4bb6dcc7e06c WatchSource:0}: Error finding container 96fae5a4031880a78bac9a755c06367fa3a998e276d66611b25f4bb6dcc7e06c: Status 404 returned error can't find the container with id 96fae5a4031880a78bac9a755c06367fa3a998e276d66611b25f4bb6dcc7e06c
Oct 30 00:41:58.373554 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.373532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:41:58.376979 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.376957    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927717eb_ca5c_4f65_8a7a_bdc83054080c.slice/crio-d8fa684a7e915bc9a5737948f2a4f9087640d6ef8ecc4db2b47580327fba7095 WatchSource:0}: Error finding container d8fa684a7e915bc9a5737948f2a4f9087640d6ef8ecc4db2b47580327fba7095: Status 404 returned error can't find the container with id d8fa684a7e915bc9a5737948f2a4f9087640d6ef8ecc4db2b47580327fba7095
Oct 30 00:41:58.382254 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:41:58.382236    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c4505b_9dd7_4044_8c27_b1f6e2d59ea6.slice/crio-d50536e56bdf06f88016b7a66ac227f253b006e76d13557570c101b66efca658 WatchSource:0}: Error finding container d50536e56bdf06f88016b7a66ac227f253b006e76d13557570c101b66efca658: Status 404 returned error can't find the container with id d50536e56bdf06f88016b7a66ac227f253b006e76d13557570c101b66efca658
Oct 30 00:41:58.384324 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.384305    2579 kuberuntime_manager.go:1707] "Unhandled Error" err="container node-ca start failed in pod node-ca-r6gln_openshift-image-registry(e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 30 00:41:58.384378 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.384335    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-image-registry/node-ca-r6gln" podUID="e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6"
Oct 30 00:41:58.633856 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.633777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:58.633856 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.633832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:58.634053 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.634007    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:58.634101 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.634068    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs podName:ce14d2cc-a9e8-49af-be37-9b8e6a433c65 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:59.634050633 +0000 UTC m=+3.142557510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs") pod "network-metrics-daemon-2q7g6" (UID: "ce14d2cc-a9e8-49af-be37-9b8e6a433c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:58.634185 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.634163    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:58.634305 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.634222    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret podName:b3bb0053-40b7-49df-9490-192e78a9b720 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:59.634207429 +0000 UTC m=+3.142714306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret") pod "global-pull-secret-syncer-nk4k6" (UID: "b3bb0053-40b7-49df-9490-192e78a9b720") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:58.735160 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:58.735121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:41:58.735360 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.735319    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:41:58.735360 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.735345    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:41:58.735360 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.735361    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rsphz for pod openshift-network-diagnostics/network-check-target-77hhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:58.735726 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:58.735432    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz podName:f69dcc85-0b97-43eb-8674-0ec1415c8dfb nodeName:}" failed. No retries permitted until 2026-10-30 00:41:59.735411044 +0000 UTC m=+3.243917926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsphz" (UniqueName: "kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz") pod "network-check-target-77hhk" (UID: "f69dcc85-0b97-43eb-8674-0ec1415c8dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:59.014509 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.014384    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerStarted","Data":"96fae5a4031880a78bac9a755c06367fa3a998e276d66611b25f4bb6dcc7e06c"}
Oct 30 00:41:59.050357 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.050315    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-skbfn" event={"ID":"7469affe-9d00-44b3-a833-4ca25b86da5c","Type":"ContainerStarted","Data":"615b46a4c54925eacc7cbf15bec37c7ad0c000436b828c0301bbab676933ecca"}
Oct 30 00:41:59.059334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.059242    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztnp7" event={"ID":"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141","Type":"ContainerStarted","Data":"9c7c2708d7db6bd2afb1525e76a4c34188d906fe37a9b22a3409f4b3a35cb186"}
Oct 30 00:41:59.070054 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.070014    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tcqrb" event={"ID":"4254ef03-935f-47d4-948a-70855cb3afb8","Type":"ContainerStarted","Data":"64e2b2035554c6d65d1cb62e61ebe849dbc427fd3a978173782c3a3df331a3d1"}
Oct 30 00:41:59.076871 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.076822    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:36:57 +0000 UTC" deadline="2028-06-16 21:15:10.727731298 +0000 UTC"
Oct 30 00:41:59.076871 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.076868    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14300h33m11.650865534s"
Oct 30 00:41:59.089474 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.089428    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rxj86" event={"ID":"7d027727-5c89-44a2-b1b8-20661fd13909","Type":"ContainerStarted","Data":"bdbb2c8206606e39d673e1c3da53ada1679b600e07a2235916f856cfc32041b6"}
Oct 30 00:41:59.095585 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.095560    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:59.102441 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.102405    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal" event={"ID":"c2831292250175cd4fa5c7c5f1d89a07","Type":"ContainerStarted","Data":"37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b"}
Oct 30 00:41:59.110170 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.110135    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r6gln" event={"ID":"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6","Type":"ContainerStarted","Data":"d50536e56bdf06f88016b7a66ac227f253b006e76d13557570c101b66efca658"}
Oct 30 00:41:59.115153 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.115120    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11509ac76e128ae597c6ac16bf61ea639d70c1a8f19835bb5fc0d4c9fcf3867c\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-image-registry/node-ca-r6gln" podUID="e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6"
Oct 30 00:41:59.117689 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.117567    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ns76" event={"ID":"927717eb-ca5c-4f65-8a7a-bdc83054080c","Type":"ContainerStarted","Data":"d8fa684a7e915bc9a5737948f2a4f9087640d6ef8ecc4db2b47580327fba7095"}
Oct 30 00:41:59.138151 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.138108    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7" event={"ID":"a8e082e1-ce44-4deb-8036-89d2ae8dcc46","Type":"ContainerStarted","Data":"501238e2345bfe4f9ebfa0008d8f4b12e0993b7905f8f4bad3f7f3393b72ccc7"}
Oct 30 00:41:59.148182 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.148147    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"303442b5c031dfa67e64696ad482f1f3466163bbf51b19d452685f0a7663048e"}
Oct 30 00:41:59.159333 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.159297    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal" event={"ID":"3bf851886b8d1abdda99ba2c15c29f78","Type":"ContainerStarted","Data":"19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d"}
Oct 30 00:41:59.642663 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.642625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:59.642663 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.642665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:59.642892 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.642795    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:59.642892 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.642841    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs podName:ce14d2cc-a9e8-49af-be37-9b8e6a433c65 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:01.642828293 +0000 UTC m=+5.151335170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs") pod "network-metrics-daemon-2q7g6" (UID: "ce14d2cc-a9e8-49af-be37-9b8e6a433c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:59.643190 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.643136    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:59.643190 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.643176    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret podName:b3bb0053-40b7-49df-9490-192e78a9b720 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:01.643165596 +0000 UTC m=+5.151672473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret") pod "global-pull-secret-syncer-nk4k6" (UID: "b3bb0053-40b7-49df-9490-192e78a9b720") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:59.743855 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.743814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:41:59.744138 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.744118    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:41:59.744207 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.744146    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:41:59.744207 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.744160    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rsphz for pod openshift-network-diagnostics/network-check-target-77hhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:59.744341 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.744219    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz podName:f69dcc85-0b97-43eb-8674-0ec1415c8dfb nodeName:}" failed. No retries permitted until 2026-10-30 00:42:01.744203771 +0000 UTC m=+5.252710633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsphz" (UniqueName: "kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz") pod "network-check-target-77hhk" (UID: "f69dcc85-0b97-43eb-8674-0ec1415c8dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:59.992181 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.992104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:41:59.992335 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.992239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:41:59.992683 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.992663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:41:59.992806 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.992784    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:41:59.992869 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:41:59.992858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:41:59.992958 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:41:59.992934    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:00.077657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:00.077613    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:36:57 +0000 UTC" deadline="2028-07-18 23:18:23.537182964 +0000 UTC"
Oct 30 00:42:00.077657 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:00.077654    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15070h36m23.459533316s"
Oct 30 00:42:01.661850 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:01.661804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:01.662320 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:01.661866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:01.662320 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.661983    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:42:01.662320 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.662046    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs podName:ce14d2cc-a9e8-49af-be37-9b8e6a433c65 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:05.662026849 +0000 UTC m=+9.170533729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs") pod "network-metrics-daemon-2q7g6" (UID: "ce14d2cc-a9e8-49af-be37-9b8e6a433c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:42:01.662490 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.662409    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:42:01.662490 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.662445    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret podName:b3bb0053-40b7-49df-9490-192e78a9b720 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:05.662435515 +0000 UTC m=+9.170942391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret") pod "global-pull-secret-syncer-nk4k6" (UID: "b3bb0053-40b7-49df-9490-192e78a9b720") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:42:01.762688 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:01.762652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:01.762830 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.762814    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:42:01.762882 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.762840    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:42:01.762882 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.762854    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rsphz for pod openshift-network-diagnostics/network-check-target-77hhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:42:01.762947 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.762915    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz podName:f69dcc85-0b97-43eb-8674-0ec1415c8dfb nodeName:}" failed. No retries permitted until 2026-10-30 00:42:05.762896472 +0000 UTC m=+9.271403351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsphz" (UniqueName: "kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz") pod "network-check-target-77hhk" (UID: "f69dcc85-0b97-43eb-8674-0ec1415c8dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:42:01.992285 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:01.992077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:01.992446 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.992416    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:01.992535 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:01.992506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:01.992598 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.992581    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:01.992654 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:01.992620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:01.992700 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:01.992680    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:02.179994 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:02.179962    2579 generic.go:356] "Generic (PLEG): container finished" podID="3bf851886b8d1abdda99ba2c15c29f78" containerID="6ddd8e6ee8ea0407bb055951087213f6692859f25a5b8f1e0a844f7e06660510" exitCode=0
Oct 30 00:42:02.180120 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:02.180059    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal" event={"ID":"3bf851886b8d1abdda99ba2c15c29f78","Type":"ContainerDied","Data":"6ddd8e6ee8ea0407bb055951087213f6692859f25a5b8f1e0a844f7e06660510"}
Oct 30 00:42:02.186216 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:02.186175    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerStarted","Data":"007d3b7c9e5235979299802e40b921d51d90388ec948ae537fa85d9a7fb7b643"}
Oct 30 00:42:03.192660 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.192577    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal" event={"ID":"3bf851886b8d1abdda99ba2c15c29f78","Type":"ContainerStarted","Data":"da89aca54a7cdfdc39808f2060333ad707928c837c2a02918bca422ab42d05d9"}
Oct 30 00:42:03.195871 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.195840    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="007d3b7c9e5235979299802e40b921d51d90388ec948ae537fa85d9a7fb7b643" exitCode=0
Oct 30 00:42:03.196000 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.195921    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerDied","Data":"007d3b7c9e5235979299802e40b921d51d90388ec948ae537fa85d9a7fb7b643"}
Oct 30 00:42:03.201094 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.201045    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-skbfn" event={"ID":"7469affe-9d00-44b3-a833-4ca25b86da5c","Type":"ContainerStarted","Data":"ed225dd34bd3d313f594f76988424ecb090031c52fa10734dc722dc8df68ec7c"}
Oct 30 00:42:03.204966 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.204937    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal" event={"ID":"c2831292250175cd4fa5c7c5f1d89a07","Type":"ContainerStarted","Data":"b30bc04a6379ceb2e5edcbf2d797e6583b182e18d4e6409b2e287d7594e9732c"}
Oct 30 00:42:03.231993 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.231929    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal" podStartSLOduration=6.231910207 podStartE2EDuration="6.231910207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:03.231612081 +0000 UTC m=+6.740118966" watchObservedRunningTime="2026-10-30 00:42:03.231910207 +0000 UTC m=+6.740417092"
Oct 30 00:42:03.315762 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.315686    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal" podStartSLOduration=6.315669808 podStartE2EDuration="6.315669808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:03.271146458 +0000 UTC m=+6.779653346" watchObservedRunningTime="2026-10-30 00:42:03.315669808 +0000 UTC m=+6.824176692"
Oct 30 00:42:03.992512 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.992441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:03.992670 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:03.992617    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:03.992722 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.992702    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:03.992800 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:03.992782    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:03.992851 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:03.992841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:03.992932 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:03.992907    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:04.212415 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:04.211514    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7" event={"ID":"a8e082e1-ce44-4deb-8036-89d2ae8dcc46","Type":"ContainerStarted","Data":"ca28cac24d0e551a6deb125a5a51d73d87af02e909fcb1d88ee9cfca218a0668"}
Oct 30 00:42:04.461880 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:04.461789    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:05.489193 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.489141    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:42:05.694666 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.694495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:05.694666 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.694558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:05.694877 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.694727    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:42:05.694877 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.694791    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs podName:ce14d2cc-a9e8-49af-be37-9b8e6a433c65 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:13.694773122 +0000 UTC m=+17.203280002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs") pod "network-metrics-daemon-2q7g6" (UID: "ce14d2cc-a9e8-49af-be37-9b8e6a433c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:42:05.695102 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.695074    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:42:05.695245 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.695145    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret podName:b3bb0053-40b7-49df-9490-192e78a9b720 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:13.69512264 +0000 UTC m=+17.203629512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret") pod "global-pull-secret-syncer-nk4k6" (UID: "b3bb0053-40b7-49df-9490-192e78a9b720") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:42:05.795402 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.795352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:05.795618 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.795493    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:42:05.795618 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.795509    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:42:05.795618 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.795519    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rsphz for pod openshift-network-diagnostics/network-check-target-77hhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:42:05.795618 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.795570    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz podName:f69dcc85-0b97-43eb-8674-0ec1415c8dfb nodeName:}" failed. No retries permitted until 2026-10-30 00:42:13.795556341 +0000 UTC m=+17.304063218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsphz" (UniqueName: "kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz") pod "network-check-target-77hhk" (UID: "f69dcc85-0b97-43eb-8674-0ec1415c8dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:42:05.983470 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.983249    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:42:05.489164734Z","UUID":"8ea917ce-e93e-4ac1-af04-82e72f081de4","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:42:05.985031 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.985006    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:42:05.985148 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.985040    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:42:05.992555 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.992527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:05.992672 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.992651    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:05.992749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.992732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:05.992847 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.992823    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:05.992916 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:05.992883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:05.992976 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:05.992955    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:06.217526 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:06.217221    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tcqrb" event={"ID":"4254ef03-935f-47d4-948a-70855cb3afb8","Type":"ContainerStarted","Data":"38736d2d34e43ed17af82635d7c8140079bc6291f3e614ec210b7c1e616ab526"}
Oct 30 00:42:06.221202 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:06.221167    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7" event={"ID":"a8e082e1-ce44-4deb-8036-89d2ae8dcc46","Type":"ContainerStarted","Data":"27203686538fb9a574b1eb0ec7233e71044ee60815845f767ab3d610aafc5a02"}
Oct 30 00:42:06.259531 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:06.259480    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-skbfn" podStartSLOduration=5.051843498 podStartE2EDuration="9.259466254s" totalImagesPullingTime="4.207622756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:03.355167704 +0000 UTC m=+6.863674591" watchObservedRunningTime="2026-10-30 00:42:06.259466254 +0000 UTC m=+9.767973130"
Oct 30 00:42:07.229684 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.225851    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rxj86" event={"ID":"7d027727-5c89-44a2-b1b8-20661fd13909","Type":"ContainerStarted","Data":"bb264927778cfc376fd12d9474f17b039422be0ba1690f4c81599bced6c29dd1"}
Oct 30 00:42:07.230792 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.230762    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7" event={"ID":"a8e082e1-ce44-4deb-8036-89d2ae8dcc46","Type":"ContainerStarted","Data":"d99d7db35aef386a2ec9791b9549f29babc3a528b85f6932e8352aa4c75ab515"}
Oct 30 00:42:07.233929 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.233886    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerStarted","Data":"4c6582d5c5677ebd9ce4950331a8c013299566ee288b5df42f1d0e742f773e32"}
Oct 30 00:42:07.267194 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.266322    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rxj86" podStartSLOduration=2.093728994 podStartE2EDuration="10.266304486s" totalImagesPullingTime="8.172575492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:07.266058944 +0000 UTC m=+10.774565830" watchObservedRunningTime="2026-10-30 00:42:07.266304486 +0000 UTC m=+10.774811377"
Oct 30 00:42:07.267194 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.266444    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tcqrb" podStartSLOduration=2.6560587829999998 podStartE2EDuration="10.266436642s" totalImagesPullingTime="7.610377859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:06.260094513 +0000 UTC m=+9.768601414" watchObservedRunningTime="2026-10-30 00:42:07.266436642 +0000 UTC m=+10.774943528"
Oct 30 00:42:07.309217 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.309150    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7" podStartSLOduration=2.2758739390000002 podStartE2EDuration="10.309130667s" totalImagesPullingTime="8.033256728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:07.308301385 +0000 UTC m=+10.816808272" watchObservedRunningTime="2026-10-30 00:42:07.309130667 +0000 UTC m=+10.817637614"
Oct 30 00:42:07.918068 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.917366    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:07.918525 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.918447    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:07.992732 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.992658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:07.992732 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.992672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:07.992928 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:07.992792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:07.992928 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:07.992805    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:07.992928 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:07.992898    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:07.993111 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:07.992973    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:08.237108 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:08.237078    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ns76" event={"ID":"927717eb-ca5c-4f65-8a7a-bdc83054080c","Type":"ContainerStarted","Data":"9b77597bff7701284847c25288dfc513fa400cadc975fc665ef034a049aa2bc7"}
Oct 30 00:42:08.239521 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:08.239491    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="4c6582d5c5677ebd9ce4950331a8c013299566ee288b5df42f1d0e742f773e32" exitCode=0
Oct 30 00:42:08.240220 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:08.239636    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerDied","Data":"4c6582d5c5677ebd9ce4950331a8c013299566ee288b5df42f1d0e742f773e32"}
Oct 30 00:42:08.240220 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:08.240197    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:08.278293 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:08.278243    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6ns76" podStartSLOduration=3.502593581 podStartE2EDuration="11.278229018s" totalImagesPullingTime="7.775635437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:08.278035968 +0000 UTC m=+11.786542855" watchObservedRunningTime="2026-10-30 00:42:08.278229018 +0000 UTC m=+11.786735902"
Oct 30 00:42:09.244918 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:09.244833    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="ccdbfc1aead2a2aaf5ae26f1f63e6b620332f252115b738a44b3b3ce3197ba38" exitCode=0
Oct 30 00:42:09.245509 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:09.244922    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerDied","Data":"ccdbfc1aead2a2aaf5ae26f1f63e6b620332f252115b738a44b3b3ce3197ba38"}
Oct 30 00:42:09.246818 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:09.246791    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztnp7" event={"ID":"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141","Type":"ContainerStarted","Data":"d93c9d31d1bc712be40472a6d42fae29fd4fc7c5f1b73b6b7612835addd1f718"}
Oct 30 00:42:09.347766 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:09.347712    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ztnp7" podStartSLOduration=2.437955088 podStartE2EDuration="12.347695587s" totalImagesPullingTime="9.909740499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:09.347494573 +0000 UTC m=+12.856001458" watchObservedRunningTime="2026-10-30 00:42:09.347695587 +0000 UTC m=+12.856202473"
Oct 30 00:42:09.992305 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:09.992274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:09.992441 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:09.992355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:09.992537 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:09.992497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:09.992537 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:09.992515    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:09.992640 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:09.992616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:09.992709 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:09.992686    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:10.252192 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:10.252097    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="a1f2be932b4253e40b54d7df954050a09f0cee352fc141d5c9f839afb482c3a7" exitCode=0
Oct 30 00:42:10.252961 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:10.252193    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerDied","Data":"a1f2be932b4253e40b54d7df954050a09f0cee352fc141d5c9f839afb482c3a7"}
Oct 30 00:42:11.993086 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:11.992866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:11.993086 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:11.993045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:11.993086 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:11.993007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:11.993609 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:11.993179    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:11.993609 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:11.993279    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:11.993609 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:11.993347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:12.259999 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:12.259886    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="11e9d572e955dd0c777d48fed519d371dbf8aa0dcb35c676d4b120704c2c88fa" exitCode=0
Oct 30 00:42:12.259999 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:12.259961    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerDied","Data":"11e9d572e955dd0c777d48fed519d371dbf8aa0dcb35c676d4b120704c2c88fa"}
Oct 30 00:42:13.265240 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.265217    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:13.265656 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.265625    2579 generic.go:356] "Generic (PLEG): container finished" podID="085c11e9-3ae8-4a1d-abe7-edc7dcad9804" containerID="09d4982670b7f504b69df58781cf46234e80f64aef14260654a5a2404674367f" exitCode=1
Oct 30 00:42:13.265717 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.265695    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerDied","Data":"09d4982670b7f504b69df58781cf46234e80f64aef14260654a5a2404674367f"}
Oct 30 00:42:13.265795 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.265776    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"68e417a9b91246f57b3727bd70c1297df908f9bce3d91e5b2f88c8a086d62297"}
Oct 30 00:42:13.269412 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.269387    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="b8d9aba6ae230089b9fee539d199dd556930b6a1000e985a1e76002436377b0a" exitCode=0
Oct 30 00:42:13.269547 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.269429    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerDied","Data":"b8d9aba6ae230089b9fee539d199dd556930b6a1000e985a1e76002436377b0a"}
Oct 30 00:42:13.752245 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.752068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:13.752448 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.752259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:13.752448 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.752221    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:42:13.752448 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.752349    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:42:13.752448 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.752388    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret podName:b3bb0053-40b7-49df-9490-192e78a9b720 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:29.752366356 +0000 UTC m=+33.260873223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret") pod "global-pull-secret-syncer-nk4k6" (UID: "b3bb0053-40b7-49df-9490-192e78a9b720") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:42:13.752448 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.752407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs podName:ce14d2cc-a9e8-49af-be37-9b8e6a433c65 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:29.75239845 +0000 UTC m=+33.260905313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs") pod "network-metrics-daemon-2q7g6" (UID: "ce14d2cc-a9e8-49af-be37-9b8e6a433c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:42:13.853492 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.853448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:13.853625 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.853594    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:42:13.853625 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.853616    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:42:13.853726 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.853629    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rsphz for pod openshift-network-diagnostics/network-check-target-77hhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:42:13.853726 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.853696    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz podName:f69dcc85-0b97-43eb-8674-0ec1415c8dfb nodeName:}" failed. No retries permitted until 2026-10-30 00:42:29.853676607 +0000 UTC m=+33.362183485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsphz" (UniqueName: "kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz") pod "network-check-target-77hhk" (UID: "f69dcc85-0b97-43eb-8674-0ec1415c8dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:42:13.992243 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.992208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:13.992440 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.992301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:13.992440 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:13.992323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:13.992440 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.992427    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:13.992635 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.992541    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:13.992635 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:13.992591    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:14.275597 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.275519    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" event={"ID":"a8d8e7da-34ee-466e-bf3d-48a8a56b722f","Type":"ContainerStarted","Data":"9b7729893b7fa477b4a439a51e6cc74761836be935603b6032c5a56924088641"}
Oct 30 00:42:14.277024 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.276999    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r6gln" event={"ID":"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6","Type":"ContainerStarted","Data":"61ad855929cafd84a25b13f735c7175d9e53adde1619f8576fd81ab3f3530265"}
Oct 30 00:42:14.279554 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.279535    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:14.279872 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.279854    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"3b0e2dd64140eaa9708499747beb4c5867afad0a1f33dec16e85fbcf12c7a0ba"}
Oct 30 00:42:14.279956 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.279913    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"9ee8e12de88e810e195b4908f2d91625a5d2868ade2ad082a8159a22c9414681"}
Oct 30 00:42:14.279956 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.279944    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"edbf60d368a6ac0435f4bbfe3962b97971d23aed7484df16998a473fb13751a9"}
Oct 30 00:42:14.280018 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.279971    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"fd529c21eb0e4ba4a4f78cdf710a7650212ba2a6eca4f7d7d58a079076386aaf"}
Oct 30 00:42:14.319765 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.319718    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b6zxp" podStartSLOduration=7.775748947 podStartE2EDuration="17.31970338s" totalImagesPullingTime="9.543954433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:14.319394888 +0000 UTC m=+17.827901783" watchObservedRunningTime="2026-10-30 00:42:14.31970338 +0000 UTC m=+17.828210264"
Oct 30 00:42:14.358514 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:14.358443    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r6gln" podStartSLOduration=1.961128432 podStartE2EDuration="17.358431031s" totalImagesPullingTime="15.397302599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-30 00:42:14.358326209 +0000 UTC m=+17.866833094" watchObservedRunningTime="2026-10-30 00:42:14.358431031 +0000 UTC m=+17.866937916"
Oct 30 00:42:15.992536 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:15.992436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:15.992963 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:15.992436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:15.992963 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:15.992565    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:15.992963 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:15.992440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:15.992963 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:15.992642    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:15.992963 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:15.992753    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:16.287305 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:16.287220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:16.287646 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:16.287616    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"a04e4f0b50f240eb22a71239c58d18b44e9653b67466a97abf8fbae31c1c640e"}
Oct 30 00:42:17.992733 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:17.992661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:17.993321 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:17.992661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:17.993321 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:17.992766    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:17.993321 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:17.992779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:17.993321 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:17.992864    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:17.993321 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:17.992943    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:18.295277 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:18.295213    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:18.295595 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:18.295569    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"63f336aebe7389b6b7d1f2d2bb45aabb9d25d71ddf981bd2a7353cfc979b7aec"}
Oct 30 00:42:18.295918 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:18.295902    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:18.296492 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:18.296468    2579 scope.go:122] "RemoveContainer" containerID="09d4982670b7f504b69df58781cf46234e80f64aef14260654a5a2404674367f"
Oct 30 00:42:18.311890 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:18.311866    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:19.301392 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.301214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:19.301795 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.301753    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" event={"ID":"085c11e9-3ae8-4a1d-abe7-edc7dcad9804","Type":"ContainerStarted","Data":"63d4de851f19e574a7b31d43241e885b433555b3a014a7175a1b49c507d9f8cf"}
Oct 30 00:42:19.301978 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.301957    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:19.302046 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.302032    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:19.317470 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.317430    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:19.398513 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.398444    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4" podStartSLOduration=7.742865326 podStartE2EDuration="22.398430309s" totalImagesPullingTime="14.655564983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:19.398140459 +0000 UTC m=+22.906647351" watchObservedRunningTime="2026-10-30 00:42:19.398430309 +0000 UTC m=+22.906937194"
Oct 30 00:42:19.726121 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.726044    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-77hhk"]
Oct 30 00:42:19.726248 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.726199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:19.726296 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:19.726280    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:19.729579 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.729548    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2q7g6"]
Oct 30 00:42:19.729745 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.729690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:19.729965 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:19.729937    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:19.730238 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.730211    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk4k6"]
Oct 30 00:42:19.730347 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:19.730326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:19.730443 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:19.730419    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:20.992129 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:20.992099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:20.992624 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:20.992208    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:21.992506 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:21.992447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:21.992908 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:21.992472    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:21.992908 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:21.992582    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:21.992908 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:21.992649    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:22.992921 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:22.992883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:22.993501 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:22.993017    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:23.992014 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:23.991982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:23.992184 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:23.991982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:23.992184 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:23.992115    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:23.992255 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:23.992197    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:24.992234 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:24.992203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:24.992656 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:24.992309    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk4k6" podUID="b3bb0053-40b7-49df-9490-192e78a9b720"
Oct 30 00:42:25.992685 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:25.992651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:25.993149 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:25.992695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:25.993149 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:25.992784    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2q7g6" podUID="ce14d2cc-a9e8-49af-be37-9b8e6a433c65"
Oct 30 00:42:25.993149 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:25.992879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77hhk" podUID="f69dcc85-0b97-43eb-8674-0ec1415c8dfb"
Oct 30 00:42:26.801039 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.801011    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeReady"
Oct 30 00:42:26.801182 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.801135    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:42:26.846475 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.846428    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ss447"]
Oct 30 00:42:26.920770 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.920740    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xf7xr"]
Oct 30 00:42:26.921299 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.920788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ss447"
Oct 30 00:42:26.933011 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.932987    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rrv72"]
Oct 30 00:42:26.933138 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.933118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xf7xr"
Oct 30 00:42:26.937649 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.937624    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 00:42:26.937767 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.937749    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 00:42:26.947126 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.947102    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"]
Oct 30 00:42:26.947223 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.947207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rrv72"
Oct 30 00:42:26.961209 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.961187    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"]
Oct 30 00:42:26.961326 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.961289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:26.977932 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.977911    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 00:42:26.978033 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.978009    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m75ds\""
Oct 30 00:42:26.985783 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.985761    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"]
Oct 30 00:42:26.985879 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:26.985841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"
Oct 30 00:42:27.010076 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.010052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.013223 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.013199    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5phkz"]
Oct 30 00:42:27.013321 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.013311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:27.026611 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.026591    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9wjf"]
Oct 30 00:42:27.026738 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.026724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.038011 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.037991    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.038135 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038029    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 00:42:27.038135 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038046    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.038135 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038076    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.038135 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038099    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.038339 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038187    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.038339 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038225    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.038339 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd285\" (UniqueName: \"kubernetes.io/projected/6a4543fe-837e-4480-b419-974b723e4475-kube-api-access-fd285\") pod \"kas-connection-checker-685895c7c9-rrv72\" (UID: \"6a4543fe-837e-4480-b419-974b723e4475\") " pod="kube-system/kas-connection-checker-685895c7c9-rrv72"
Oct 30 00:42:27.038339 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.038569 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ncq\" (UniqueName: \"kubernetes.io/projected/79ef76c0-8e88-4b4a-8d6a-83da18b39d78-kube-api-access-55ncq\") pod \"kas-connection-checker-685895c7c9-xf7xr\" (UID: \"79ef76c0-8e88-4b4a-8d6a-83da18b39d78\") " pod="kube-system/kas-connection-checker-685895c7c9-xf7xr"
Oct 30 00:42:27.038569 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pw8p\" (UniqueName: \"kubernetes.io/projected/2deae516-c4e0-4377-98ee-084581c4b027-kube-api-access-5pw8p\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.038677 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnf6\" (UniqueName: \"kubernetes.io/projected/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405-kube-api-access-4cnf6\") pod \"kas-connection-checker-685895c7c9-ss447\" (UID: \"2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405\") " pod="kube-system/kas-connection-checker-685895c7c9-ss447"
Oct 30 00:42:27.038677 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038600    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 00:42:27.038783 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/9cec9dfe-c016-4228-be90-4724120f7260-kube-api-access-z546g\") pod \"volume-data-source-validator-58c9f5f46d-56c4z\" (UID: \"9cec9dfe-c016-4228-be90-4724120f7260\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"
Oct 30 00:42:27.038783 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.038715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2deae516-c4e0-4377-98ee-084581c4b027-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.050845 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.050825    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"]
Oct 30 00:42:27.050963 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.050946    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.073837 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.073810    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54566cf8bb-wxl8f"]
Oct 30 00:42:27.073954 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.073924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.078516 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078495    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 00:42:27.078639 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078513    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 00:42:27.078736 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078639    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7ssb7\""
Oct 30 00:42:27.078736 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078713    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-chgph\""
Oct 30 00:42:27.078881 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078740    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:42:27.078881 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078701    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 00:42:27.078881 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078688    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cw5zq\""
Oct 30 00:42:27.078881 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078761    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\""
Oct 30 00:42:27.078881 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.078840    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-v9ptc\""
Oct 30 00:42:27.098742 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.098720    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75647bc475-s8spf"]
Oct 30 00:42:27.098839 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.098764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.114787 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.114765    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-s248j"]
Oct 30 00:42:27.114906 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.114883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.129021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.129001    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7488dfd947-qp2j9"]
Oct 30 00:42:27.129120 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.129105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:27.138636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138616    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:42:27.138636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138623    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138696    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138700    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138747    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138756    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138805    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138813    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138821    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 00:42:27.138863 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.138844    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139005    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5tx8\" (UniqueName: \"kubernetes.io/projected/b9dac87d-7a94-43b5-ae69-38f6975388ed-kube-api-access-d5tx8\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt8t\" (UniqueName: \"kubernetes.io/projected/81da025c-1894-43c8-b83a-0b80409656ac-kube-api-access-5dt8t\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-config\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dac87d-7a94-43b5-ae69-38f6975388ed-serving-cert\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.139148    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ncq\" (UniqueName: \"kubernetes.io/projected/79ef76c0-8e88-4b4a-8d6a-83da18b39d78-kube-api-access-55ncq\") pod \"kas-connection-checker-685895c7c9-xf7xr\" (UID: \"79ef76c0-8e88-4b4a-8d6a-83da18b39d78\") " pod="kube-system/kas-connection-checker-685895c7c9-xf7xr"
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.139197    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls podName:2deae516-c4e0-4377-98ee-084581c4b027 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.639178111 +0000 UTC m=+31.147685012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vfg9g" (UID: "2deae516-c4e0-4377-98ee-084581c4b027") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:27.139279 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139228    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9qt\" (UniqueName: \"kubernetes.io/projected/33045ca4-6bda-4fdc-9d44-f231e4161fae-kube-api-access-sl9qt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33045ca4-6bda-4fdc-9d44-f231e4161fae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pw8p\" (UniqueName: \"kubernetes.io/projected/2deae516-c4e0-4377-98ee-084581c4b027-kube-api-access-5pw8p\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnf6\" (UniqueName: \"kubernetes.io/projected/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405-kube-api-access-4cnf6\") pod \"kas-connection-checker-685895c7c9-ss447\" (UID: \"2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405\") " pod="kube-system/kas-connection-checker-685895c7c9-ss447"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-trusted-ca\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-snapshots\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33045ca4-6bda-4fdc-9d44-f231e4161fae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gql2j\" (UniqueName: \"kubernetes.io/projected/0240e78b-bcf9-40a0-8707-0f7976d91005-kube-api-access-gql2j\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/9cec9dfe-c016-4228-be90-4724120f7260-kube-api-access-z546g\") pod \"volume-data-source-validator-58c9f5f46d-56c4z\" (UID: \"9cec9dfe-c016-4228-be90-4724120f7260\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"
Oct 30 00:42:27.139948 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-tmp\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.140356 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2deae516-c4e0-4377-98ee-084581c4b027-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.140356 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.139992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81da025c-1894-43c8-b83a-0b80409656ac-serving-cert\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.140356 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.140013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd285\" (UniqueName: \"kubernetes.io/projected/6a4543fe-837e-4480-b419-974b723e4475-kube-api-access-fd285\") pod \"kas-connection-checker-685895c7c9-rrv72\" (UID: \"6a4543fe-837e-4480-b419-974b723e4475\") " pod="kube-system/kas-connection-checker-685895c7c9-rrv72"
Oct 30 00:42:27.140532 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.140515    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2deae516-c4e0-4377-98ee-084581c4b027-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.154694 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.154665    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 00:42:27.154995 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.154973    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 00:42:27.155169 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.155105    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:42:27.155214 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.155196    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:42:27.159196 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.159180    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"]
Oct 30 00:42:27.159296 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.159283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.170306 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.170284    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ncq\" (UniqueName: \"kubernetes.io/projected/79ef76c0-8e88-4b4a-8d6a-83da18b39d78-kube-api-access-55ncq\") pod \"kas-connection-checker-685895c7c9-xf7xr\" (UID: \"79ef76c0-8e88-4b4a-8d6a-83da18b39d78\") " pod="kube-system/kas-connection-checker-685895c7c9-xf7xr"
Oct 30 00:42:27.170419 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.170323    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/9cec9dfe-c016-4228-be90-4724120f7260-kube-api-access-z546g\") pod \"volume-data-source-validator-58c9f5f46d-56c4z\" (UID: \"9cec9dfe-c016-4228-be90-4724120f7260\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"
Oct 30 00:42:27.170419 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.170333    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnf6\" (UniqueName: \"kubernetes.io/projected/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405-kube-api-access-4cnf6\") pod \"kas-connection-checker-685895c7c9-ss447\" (UID: \"2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405\") " pod="kube-system/kas-connection-checker-685895c7c9-ss447"
Oct 30 00:42:27.170553 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.170494    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pw8p\" (UniqueName: \"kubernetes.io/projected/2deae516-c4e0-4377-98ee-084581c4b027-kube-api-access-5pw8p\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.170855 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.170836    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd285\" (UniqueName: \"kubernetes.io/projected/6a4543fe-837e-4480-b419-974b723e4475-kube-api-access-fd285\") pod \"kas-connection-checker-685895c7c9-rrv72\" (UID: \"6a4543fe-837e-4480-b419-974b723e4475\") " pod="kube-system/kas-connection-checker-685895c7c9-rrv72"
Oct 30 00:42:27.173421 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.173405    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"]
Oct 30 00:42:27.173563 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.173545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"
Oct 30 00:42:27.178446 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178424    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 00:42:27.178544 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178449    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 00:42:27.178544 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178510    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 00:42:27.178654 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178559    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 00:42:27.178654 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178517    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:42:27.178740 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178660    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 00:42:27.178740 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178679    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:42:27.178809 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.178747    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5nrg4\""
Oct 30 00:42:27.179116 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.179102    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2cb6h\""
Oct 30 00:42:27.179207 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.179192    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fmgv7\""
Oct 30 00:42:27.179261 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.179220    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wp9zh\""
Oct 30 00:42:27.179261 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.179236    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:42:27.189649 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.189635    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hrfmc\""
Oct 30 00:42:27.194953 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.194930    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dkpn9"]
Oct 30 00:42:27.195076 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.195058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.215715 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.215688    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-26wcv"]
Oct 30 00:42:27.215812 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.215800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.237994 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.237968    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ss447"]
Oct 30 00:42:27.238104 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238005    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xf7xr"]
Oct 30 00:42:27.238104 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238028    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rrv72"]
Oct 30 00:42:27.238104 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238049    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"]
Oct 30 00:42:27.238104 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238072    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5phkz"]
Oct 30 00:42:27.238104 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.238104 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238094    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9wjf"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238117    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238146    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75647bc475-s8spf"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238169    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7488dfd947-qp2j9"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238192    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-s248j"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238214    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dkpn9"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238245    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238298    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"]
Oct 30 00:42:27.238334 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238334    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54566cf8bb-wxl8f"]
Oct 30 00:42:27.238719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238358    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"]
Oct 30 00:42:27.238719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238376    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26wcv"]
Oct 30 00:42:27.238719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238390    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"]
Oct 30 00:42:27.238719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238703    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.238719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238711    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.238719 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238719    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.238944 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238743    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.238944 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238753    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:42:27.238944 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.238800    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 00:42:27.240542 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjfv\" (UniqueName: \"kubernetes.io/projected/db543397-1d27-4098-a794-7a57aaa7904b-kube-api-access-ctjfv\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.240596 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.240633 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.240675 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tx8\" (UniqueName: \"kubernetes.io/projected/b9dac87d-7a94-43b5-ae69-38f6975388ed-kube-api-access-d5tx8\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.240710 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.240742 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.240811 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:27.240855 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-stats-auth\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.240898 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.240938 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-config\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.240938 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.240934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dac87d-7a94-43b5-ae69-38f6975388ed-serving-cert\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.241018 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfzl\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.241108 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.241172 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.241216 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33045ca4-6bda-4fdc-9d44-f231e4161fae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.241249 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-trusted-ca\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.241298 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.241298 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85585df9-d495-4d44-bd82-398d44b76cf8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:27.241393 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.241446 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gql2j\" (UniqueName: \"kubernetes.io/projected/0240e78b-bcf9-40a0-8707-0f7976d91005-kube-api-access-gql2j\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.241446 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33045ca4-6bda-4fdc-9d44-f231e4161fae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.241567 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.241567 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-tmp\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.241567 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81da025c-1894-43c8-b83a-0b80409656ac-serving-cert\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.241711 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db543397-1d27-4098-a794-7a57aaa7904b-serving-cert\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.241711 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.241711 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzt92\" (UniqueName: \"kubernetes.io/projected/44aeb453-223b-4cfa-a838-6798b6b5aa1c-kube-api-access-fzt92\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.241711 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.241711 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qq9\" (UniqueName: \"kubernetes.io/projected/56ea1b39-0a84-4b31-a200-0c1b68cfc927-kube-api-access-w7qq9\") pod \"network-check-source-6f78d59f99-fkqq5\" (UID: \"56ea1b39-0a84-4b31-a200-0c1b68cfc927\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"
Oct 30 00:42:27.241940 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241706    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-config\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.241940 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241725    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.241940 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.241730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.241940 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.241927    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:42:27.242165 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.241990    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls podName:0240e78b-bcf9-40a0-8707-0f7976d91005 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.741967718 +0000 UTC m=+31.250474593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n7f4l" (UID: "0240e78b-bcf9-40a0-8707-0f7976d91005") : secret "samples-operator-tls" not found
Oct 30 00:42:27.242165 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242034    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-tmp\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.242165 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.242331 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt8t\" (UniqueName: \"kubernetes.io/projected/81da025c-1894-43c8-b83a-0b80409656ac-kube-api-access-5dt8t\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.242331 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.242519 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.242624 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242529    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33045ca4-6bda-4fdc-9d44-f231e4161fae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.242722 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.242802 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9qt\" (UniqueName: \"kubernetes.io/projected/33045ca4-6bda-4fdc-9d44-f231e4161fae-kube-api-access-sl9qt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.242857 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlj97\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.242857 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-default-certificate\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.242965 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.242965 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db543397-1d27-4098-a794-7a57aaa7904b-config\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.242965 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-snapshots\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.242965 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.243147 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.242986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.243193 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.243142    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-trusted-ca\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.243659 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.243625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33045ca4-6bda-4fdc-9d44-f231e4161fae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.243784 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.243765    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dac87d-7a94-43b5-ae69-38f6975388ed-serving-cert\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.243869 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.243798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.243936 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.243892    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-snapshots\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.244141 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.244121    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.244198 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.244127    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81da025c-1894-43c8-b83a-0b80409656ac-serving-cert\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.255850 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.255833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ss447"
Oct 30 00:42:27.260476 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.260428    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tx8\" (UniqueName: \"kubernetes.io/projected/b9dac87d-7a94-43b5-ae69-38f6975388ed-kube-api-access-d5tx8\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.262167 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.262145    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gql2j\" (UniqueName: \"kubernetes.io/projected/0240e78b-bcf9-40a0-8707-0f7976d91005-kube-api-access-gql2j\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.262970 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.262945    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt8t\" (UniqueName: \"kubernetes.io/projected/81da025c-1894-43c8-b83a-0b80409656ac-kube-api-access-5dt8t\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.263040 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.262979    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9qt\" (UniqueName: \"kubernetes.io/projected/33045ca4-6bda-4fdc-9d44-f231e4161fae-kube-api-access-sl9qt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.265898 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.265880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xf7xr"
Oct 30 00:42:27.279295 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.279276    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gz8r\""
Oct 30 00:42:27.279395 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.279337    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ph2zs\""
Oct 30 00:42:27.279395 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.279343    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 00:42:27.279395 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.279367    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:42:27.279533 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.279403    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:42:27.279533 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.279446    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-smdbv\""
Oct 30 00:42:27.296351 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.296331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rrv72"
Oct 30 00:42:27.339023 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.338843    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:42:27.339023 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.338843    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:42:27.341699 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.341673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"
Oct 30 00:42:27.344025 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.344078 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.344111 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:27.344111 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.344167 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-stats-auth\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.344167 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.344226 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfzl\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.344226 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.344284 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.344284 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85585df9-d495-4d44-bd82-398d44b76cf8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:27.344344 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp67x\" (UniqueName: \"kubernetes.io/projected/49fcf677-f503-492a-90e8-c95e329ee6f6-kube-api-access-wp67x\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.344438 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.344438 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db543397-1d27-4098-a794-7a57aaa7904b-serving-cert\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.344575 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.344575 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzt92\" (UniqueName: \"kubernetes.io/projected/44aeb453-223b-4cfa-a838-6798b6b5aa1c-kube-api-access-fzt92\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.344575 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.344575 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qq9\" (UniqueName: \"kubernetes.io/projected/56ea1b39-0a84-4b31-a200-0c1b68cfc927-kube-api-access-w7qq9\") pod \"network-check-source-6f78d59f99-fkqq5\" (UID: \"56ea1b39-0a84-4b31-a200-0c1b68cfc927\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"
Oct 30 00:42:27.344754 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.344754 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59071108-f1ba-4346-beab-5cb766aecede-cert\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.344754 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.344754 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.344754 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29q9f\" (UniqueName: \"kubernetes.io/projected/59071108-f1ba-4346-beab-5cb766aecede-kube-api-access-29q9f\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.344754 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlj97\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-default-certificate\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db543397-1d27-4098-a794-7a57aaa7904b-config\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fcf677-f503-492a-90e8-c95e329ee6f6-tmp-dir\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.344961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49fcf677-f503-492a-90e8-c95e329ee6f6-config-volume\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.345021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.345006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjfv\" (UniqueName: \"kubernetes.io/projected/db543397-1d27-4098-a794-7a57aaa7904b-kube-api-access-ctjfv\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.345388 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.345031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.345388 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.345056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.346763 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.346113    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:42:27.346763 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.346197    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert podName:85585df9-d495-4d44-bd82-398d44b76cf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.846173978 +0000 UTC m=+31.354680856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s248j" (UID: "85585df9-d495-4d44-bd82-398d44b76cf8") : secret "networking-console-plugin-cert" not found
Oct 30 00:42:27.348292 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.348169    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.349485 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.349183    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.349485 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.349314    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.349485 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.349401    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:27.349485 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.349416    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:27.349485 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.349442    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.349485 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.349481    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.849446862 +0000 UTC m=+31.357953745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:27.349971 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.349949    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.350035 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.349955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-default-certificate\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.350096 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.350075    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:27.350096 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.350095    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:27.350195 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.350097    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85585df9-d495-4d44-bd82-398d44b76cf8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:27.350195 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.350134    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.850121141 +0000 UTC m=+31.358628018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:27.350300 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.350229    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.850217692 +0000 UTC m=+31.358724590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:27.350300 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.350257    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:42:27.350404 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.350320    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.85030404 +0000 UTC m=+31.358810903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : secret "router-metrics-certs-default" not found
Oct 30 00:42:27.350824 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.350798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db543397-1d27-4098-a794-7a57aaa7904b-config\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.351663 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.351640    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.353522 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.353181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.353981 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.353958    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.354661 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.354642    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.354735 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.354699    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-stats-auth\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.355256 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.355228    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.356487 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.355555    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db543397-1d27-4098-a794-7a57aaa7904b-serving-cert\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.373668 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.373589    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qq9\" (UniqueName: \"kubernetes.io/projected/56ea1b39-0a84-4b31-a200-0c1b68cfc927-kube-api-access-w7qq9\") pod \"network-check-source-6f78d59f99-fkqq5\" (UID: \"56ea1b39-0a84-4b31-a200-0c1b68cfc927\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"
Oct 30 00:42:27.374866 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.374837    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfzl\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.375284 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.375257    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzt92\" (UniqueName: \"kubernetes.io/projected/44aeb453-223b-4cfa-a838-6798b6b5aa1c-kube-api-access-fzt92\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.375745 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.375720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.376613 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.376588    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlj97\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.377642 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.377620    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.377866 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.377844    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjfv\" (UniqueName: \"kubernetes.io/projected/db543397-1d27-4098-a794-7a57aaa7904b-kube-api-access-ctjfv\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.388166 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.387772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:27.410959 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.409298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:27.427342 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.427102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.447123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp67x\" (UniqueName: \"kubernetes.io/projected/49fcf677-f503-492a-90e8-c95e329ee6f6-kube-api-access-wp67x\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.447223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59071108-f1ba-4346-beab-5cb766aecede-cert\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.447253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q9f\" (UniqueName: \"kubernetes.io/projected/59071108-f1ba-4346-beab-5cb766aecede-kube-api-access-29q9f\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.447331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fcf677-f503-492a-90e8-c95e329ee6f6-tmp-dir\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.447373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49fcf677-f503-492a-90e8-c95e329ee6f6-config-volume\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.447446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.447596    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:42:27.449490 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.447658    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls podName:49fcf677-f503-492a-90e8-c95e329ee6f6 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.947641777 +0000 UTC m=+31.456148640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls") pod "dns-default-dkpn9" (UID: "49fcf677-f503-492a-90e8-c95e329ee6f6") : secret "dns-default-metrics-tls" not found
Oct 30 00:42:27.455673 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.451907    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59071108-f1ba-4346-beab-5cb766aecede-cert\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.455673 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.452711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49fcf677-f503-492a-90e8-c95e329ee6f6-config-volume\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.456060 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.456013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fcf677-f503-492a-90e8-c95e329ee6f6-tmp-dir\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.468115 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.468064    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp67x\" (UniqueName: \"kubernetes.io/projected/49fcf677-f503-492a-90e8-c95e329ee6f6-kube-api-access-wp67x\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.473047 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.473020    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q9f\" (UniqueName: \"kubernetes.io/projected/59071108-f1ba-4346-beab-5cb766aecede-kube-api-access-29q9f\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.581285 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.581215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"
Oct 30 00:42:27.599829 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.599800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:27.648636 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.648595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:27.648851 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.648829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:27.649080 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.649064    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:27.649161 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.649113    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls podName:2deae516-c4e0-4377-98ee-084581c4b027 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.649101248 +0000 UTC m=+32.157608124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vfg9g" (UID: "2deae516-c4e0-4377-98ee-084581c4b027") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:27.674389 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.673684    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"]
Oct 30 00:42:27.677809 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:27.677778    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cec9dfe_c016_4228_be90_4724120f7260.slice/crio-7484c3c21e65844f4ea8b48d76affa6dc60ed06e922e34be59b4f17c2b124071 WatchSource:0}: Error finding container 7484c3c21e65844f4ea8b48d76affa6dc60ed06e922e34be59b4f17c2b124071: Status 404 returned error can't find the container with id 7484c3c21e65844f4ea8b48d76affa6dc60ed06e922e34be59b4f17c2b124071
Oct 30 00:42:27.683502 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.683470    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xf7xr"]
Oct 30 00:42:27.705421 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.705395    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ss447"]
Oct 30 00:42:27.714662 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.712751    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rrv72"]
Oct 30 00:42:27.751618 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.750172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:27.755295 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.751300    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:42:27.755295 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.755067    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls podName:0240e78b-bcf9-40a0-8707-0f7976d91005 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.755042121 +0000 UTC m=+32.263549001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n7f4l" (UID: "0240e78b-bcf9-40a0-8707-0f7976d91005") : secret "samples-operator-tls" not found
Oct 30 00:42:27.756573 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:27.756527    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81da025c_1894_43c8_b83a_0b80409656ac.slice/crio-532cd48be8aa6b55ce672db105d7bf6c304417708938946b2921e4ca89175d68 WatchSource:0}: Error finding container 532cd48be8aa6b55ce672db105d7bf6c304417708938946b2921e4ca89175d68: Status 404 returned error can't find the container with id 532cd48be8aa6b55ce672db105d7bf6c304417708938946b2921e4ca89175d68
Oct 30 00:42:27.768540 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.768411    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5phkz"]
Oct 30 00:42:27.768540 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.768484    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9wjf"]
Oct 30 00:42:27.773120 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.772591    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"]
Oct 30 00:42:27.775621 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:27.775593    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33045ca4_6bda_4fdc_9d44_f231e4161fae.slice/crio-586e2881c8d32bdea15368b7446c8b0d8724e12051d75c475edefbf039d431b3 WatchSource:0}: Error finding container 586e2881c8d32bdea15368b7446c8b0d8724e12051d75c475edefbf039d431b3: Status 404 returned error can't find the container with id 586e2881c8d32bdea15368b7446c8b0d8724e12051d75c475edefbf039d431b3
Oct 30 00:42:27.851146 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.851125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.851215 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.851184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:27.851252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.851225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:27.851282 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.851262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:27.851311 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851296    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:42:27.851311 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.851305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:27.851367 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.851333259 +0000 UTC m=+32.359840139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : secret "router-metrics-certs-default" not found
Oct 30 00:42:27.851405 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851397    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:42:27.851433 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851404    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.851393655 +0000 UTC m=+32.359900539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:27.851433 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851420    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert podName:85585df9-d495-4d44-bd82-398d44b76cf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.851412441 +0000 UTC m=+32.359919339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s248j" (UID: "85585df9-d495-4d44-bd82-398d44b76cf8") : secret "networking-console-plugin-cert" not found
Oct 30 00:42:27.851520 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851479    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:27.851520 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851483    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:27.851520 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851488    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:27.851520 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851498    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:27.851520 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851517    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.851510575 +0000 UTC m=+32.360017437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:27.851660 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.851533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.85152116 +0000 UTC m=+32.360028050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:27.918252 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.918224    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"]
Oct 30 00:42:27.919379 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:27.919348    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ea1b39_0a84_4b31_a200_0c1b68cfc927.slice/crio-8d1037f0ef4eb537f025b0e04f1a7a749ee07d715e4ff5880c66056112dbc727 WatchSource:0}: Error finding container 8d1037f0ef4eb537f025b0e04f1a7a749ee07d715e4ff5880c66056112dbc727: Status 404 returned error can't find the container with id 8d1037f0ef4eb537f025b0e04f1a7a749ee07d715e4ff5880c66056112dbc727
Oct 30 00:42:27.947553 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.947531    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"]
Oct 30 00:42:27.948386 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:27.948360    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb543397_1d27_4098_a794_7a57aaa7904b.slice/crio-3a4e1b78a880e1c90c59e2af2ac50f73b75de77b0ad0cd58c817ac5dfa706d75 WatchSource:0}: Error finding container 3a4e1b78a880e1c90c59e2af2ac50f73b75de77b0ad0cd58c817ac5dfa706d75: Status 404 returned error can't find the container with id 3a4e1b78a880e1c90c59e2af2ac50f73b75de77b0ad0cd58c817ac5dfa706d75
Oct 30 00:42:27.952278 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.952257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:27.952415 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.952399    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:42:27.952482 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:27.952451    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls podName:49fcf677-f503-492a-90e8-c95e329ee6f6 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:28.952438065 +0000 UTC m=+32.460944928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls") pod "dns-default-dkpn9" (UID: "49fcf677-f503-492a-90e8-c95e329ee6f6") : secret "dns-default-metrics-tls" not found
Oct 30 00:42:27.979711 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.979690    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26wcv"]
Oct 30 00:42:27.981037 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:27.981006    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59071108_f1ba_4346_beab_5cb766aecede.slice/crio-ccc5ef2e03bbc6dbb03dc349ae32c22dcb1a465242ba95c8786edd7f44f44ac3 WatchSource:0}: Error finding container ccc5ef2e03bbc6dbb03dc349ae32c22dcb1a465242ba95c8786edd7f44f44ac3: Status 404 returned error can't find the container with id ccc5ef2e03bbc6dbb03dc349ae32c22dcb1a465242ba95c8786edd7f44f44ac3
Oct 30 00:42:27.991897 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.991878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:27.992097 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:27.992075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:28.008112 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.008092    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:42:28.008195 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.008169    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\""
Oct 30 00:42:28.008305 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.008205    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\""
Oct 30 00:42:28.327375 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.327302    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26wcv" event={"ID":"59071108-f1ba-4346-beab-5cb766aecede","Type":"ContainerStarted","Data":"ccc5ef2e03bbc6dbb03dc349ae32c22dcb1a465242ba95c8786edd7f44f44ac3"}
Oct 30 00:42:28.328878 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.328845    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f" event={"ID":"db543397-1d27-4098-a794-7a57aaa7904b","Type":"ContainerStarted","Data":"3a4e1b78a880e1c90c59e2af2ac50f73b75de77b0ad0cd58c817ac5dfa706d75"}
Oct 30 00:42:28.331389 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.331356    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rrv72" event={"ID":"6a4543fe-837e-4480-b419-974b723e4475","Type":"ContainerStarted","Data":"296fb0cf84dc7b913a6f7eeca23e8ce7419bb13962f605f849e1a88468ca01a3"}
Oct 30 00:42:28.331532 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.331410    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rrv72" event={"ID":"6a4543fe-837e-4480-b419-974b723e4475","Type":"ContainerStarted","Data":"6cb4d9d883799c0de98745bc1149013e88d45d035c0c616121bb0b60ea1c0d23"}
Oct 30 00:42:28.332831 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.332805    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5" event={"ID":"56ea1b39-0a84-4b31-a200-0c1b68cfc927","Type":"ContainerStarted","Data":"8d1037f0ef4eb537f025b0e04f1a7a749ee07d715e4ff5880c66056112dbc727"}
Oct 30 00:42:28.334265 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.334240    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" event={"ID":"b9dac87d-7a94-43b5-ae69-38f6975388ed","Type":"ContainerStarted","Data":"6182d2c6e3c810fa7d0495e0e45b24f1383e7042c185a72d40390c9f5e88668e"}
Oct 30 00:42:28.335757 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.335730    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r" event={"ID":"33045ca4-6bda-4fdc-9d44-f231e4161fae","Type":"ContainerStarted","Data":"586e2881c8d32bdea15368b7446c8b0d8724e12051d75c475edefbf039d431b3"}
Oct 30 00:42:28.337152 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.337126    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wjf" event={"ID":"81da025c-1894-43c8-b83a-0b80409656ac","Type":"ContainerStarted","Data":"532cd48be8aa6b55ce672db105d7bf6c304417708938946b2921e4ca89175d68"}
Oct 30 00:42:28.338571 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.338549    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z" event={"ID":"9cec9dfe-c016-4228-be90-4724120f7260","Type":"ContainerStarted","Data":"7484c3c21e65844f4ea8b48d76affa6dc60ed06e922e34be59b4f17c2b124071"}
Oct 30 00:42:28.340491 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.340450    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xf7xr" event={"ID":"79ef76c0-8e88-4b4a-8d6a-83da18b39d78","Type":"ContainerStarted","Data":"3e6127e670d12d52daeb67df725bf6974bfac49c7a8452fd25aa45310da421f0"}
Oct 30 00:42:28.340575 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.340519    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xf7xr" event={"ID":"79ef76c0-8e88-4b4a-8d6a-83da18b39d78","Type":"ContainerStarted","Data":"ebee0ab5c207b2661616886648299b32db2db9f14b4ce0f50312e859067fcc96"}
Oct 30 00:42:28.343647 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.343618    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ss447" event={"ID":"2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405","Type":"ContainerStarted","Data":"c04bcfeb33e6c917c95bbeeae814a0e03592aab52a8484de2d15af6993036069"}
Oct 30 00:42:28.343730 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.343664    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ss447" event={"ID":"2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405","Type":"ContainerStarted","Data":"9063c68f92d2c33f53d770a9dedc6c9e0372507b62d00dda32a636db5999e5aa"}
Oct 30 00:42:28.371953 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.371886    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rrv72" podStartSLOduration=425.371866729 podStartE2EDuration="7m5.371866729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:28.371312383 +0000 UTC m=+31.879819272" watchObservedRunningTime="2026-10-30 00:42:28.371866729 +0000 UTC m=+31.880373615"
Oct 30 00:42:28.411787 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.411722    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xf7xr" podStartSLOduration=425.411702454 podStartE2EDuration="7m5.411702454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:28.411109348 +0000 UTC m=+31.919616232" watchObservedRunningTime="2026-10-30 00:42:28.411702454 +0000 UTC m=+31.920209340"
Oct 30 00:42:28.660003 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.659918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:28.660200 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.660183    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:28.660261 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.660255    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls podName:2deae516-c4e0-4377-98ee-084581c4b027 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.660237433 +0000 UTC m=+34.168744312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vfg9g" (UID: "2deae516-c4e0-4377-98ee-084581c4b027") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:28.762174 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.762120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:28.762423 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.762402    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:42:28.762507 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.762497    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls podName:0240e78b-bcf9-40a0-8707-0f7976d91005 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.762477962 +0000 UTC m=+34.270984840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n7f4l" (UID: "0240e78b-bcf9-40a0-8707-0f7976d91005") : secret "samples-operator-tls" not found
Oct 30 00:42:28.863704 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.863664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:28.863890 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.863747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:28.863890 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.863817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:28.863890 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.863863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:28.864047 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.863919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:28.864099 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864070    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:28.864099 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864084    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:28.864198 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.864127706 +0000 UTC m=+34.372634585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:28.864596 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864579    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.864561976 +0000 UTC m=+34.373068854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:28.864679 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864615    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:42:28.864679 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864657    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:28.864679 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864668    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:28.864797 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864703    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert podName:85585df9-d495-4d44-bd82-398d44b76cf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.864686147 +0000 UTC m=+34.373193023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s248j" (UID: "85585df9-d495-4d44-bd82-398d44b76cf8") : secret "networking-console-plugin-cert" not found
Oct 30 00:42:28.864797 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864723    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.864716469 +0000 UTC m=+34.373223332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:28.864797 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864749    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:42:28.864797 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.864786    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.864774182 +0000 UTC m=+34.373281059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : secret "router-metrics-certs-default" not found
Oct 30 00:42:28.966245 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:28.965441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:28.966245 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.965856    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:42:28.966245 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:28.965916    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls podName:49fcf677-f503-492a-90e8-c95e329ee6f6 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:30.965898515 +0000 UTC m=+34.474405395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls") pod "dns-default-dkpn9" (UID: "49fcf677-f503-492a-90e8-c95e329ee6f6") : secret "dns-default-metrics-tls" not found
Oct 30 00:42:29.775862 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:29.775825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:29.776238 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:29.775890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:29.776238 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:29.776054    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:42:29.776238 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:29.776170    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs podName:ce14d2cc-a9e8-49af-be37-9b8e6a433c65 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:01.776150499 +0000 UTC m=+65.284657375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs") pod "network-metrics-daemon-2q7g6" (UID: "ce14d2cc-a9e8-49af-be37-9b8e6a433c65") : secret "metrics-daemon-secret" not found
Oct 30 00:42:29.779021 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:29.778992    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:29.878050 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:29.877273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:29.881201 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:29.881173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:30.066141 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.066055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:30.143195 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.143134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:30.354125 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.354043    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z" event={"ID":"9cec9dfe-c016-4228-be90-4724120f7260","Type":"ContainerStarted","Data":"5b084e06bf9ccfd3c912b375dc6322005b5309778b3d4719fb8d337ec3f1c24a"}
Oct 30 00:42:30.395779 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.395723    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ss447" podStartSLOduration=427.39570534 podStartE2EDuration="7m7.39570534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:28.452820706 +0000 UTC m=+31.961327592" watchObservedRunningTime="2026-10-30 00:42:30.39570534 +0000 UTC m=+33.904212227"
Oct 30 00:42:30.396364 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.396334    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z" podStartSLOduration=400.389607402 podStartE2EDuration="6m42.396326002s" totalImagesPullingTime="2.0067186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:30.395659014 +0000 UTC m=+33.904165900" watchObservedRunningTime="2026-10-30 00:42:30.396326002 +0000 UTC m=+33.904832887"
Oct 30 00:42:30.430074 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.430044    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk4k6"]
Oct 30 00:42:30.492797 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.492767    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-77hhk"]
Oct 30 00:42:30.496531 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:30.496434    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69dcc85_0b97_43eb_8674_0ec1415c8dfb.slice/crio-8b819267f7bb1185abf55922a2797864493555e9eff9158feb54b2e7bed5057e WatchSource:0}: Error finding container 8b819267f7bb1185abf55922a2797864493555e9eff9158feb54b2e7bed5057e: Status 404 returned error can't find the container with id 8b819267f7bb1185abf55922a2797864493555e9eff9158feb54b2e7bed5057e
Oct 30 00:42:30.687805 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.687770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:30.688096 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.688014    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:30.688096 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.688080    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls podName:2deae516-c4e0-4377-98ee-084581c4b027 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.688060042 +0000 UTC m=+38.196566920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vfg9g" (UID: "2deae516-c4e0-4377-98ee-084581c4b027") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:30.789829 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.788971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:30.789829 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.789290    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:42:30.789829 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.789362    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls podName:0240e78b-bcf9-40a0-8707-0f7976d91005 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.789343652 +0000 UTC m=+38.297850540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n7f4l" (UID: "0240e78b-bcf9-40a0-8707-0f7976d91005") : secret "samples-operator-tls" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.890505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.890573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.890635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.890687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.890748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.890897    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.890913    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.890973    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.890953635 +0000 UTC m=+38.399460577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891414    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.891398883 +0000 UTC m=+38.399905772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891509    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891545    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.891533965 +0000 UTC m=+38.400040843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : secret "router-metrics-certs-default" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891614    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891624    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891664    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.891653053 +0000 UTC m=+38.400159919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891734    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:42:30.891808 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.891769    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert podName:85585df9-d495-4d44-bd82-398d44b76cf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.891757948 +0000 UTC m=+38.400264812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s248j" (UID: "85585df9-d495-4d44-bd82-398d44b76cf8") : secret "networking-console-plugin-cert" not found
Oct 30 00:42:30.991644 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:30.991549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:30.991801 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.991726    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:42:30.991867 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:30.991801    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls podName:49fcf677-f503-492a-90e8-c95e329ee6f6 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.991771229 +0000 UTC m=+38.500278092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls") pod "dns-default-dkpn9" (UID: "49fcf677-f503-492a-90e8-c95e329ee6f6") : secret "dns-default-metrics-tls" not found
Oct 30 00:42:31.360577 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:31.360483    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nk4k6" event={"ID":"b3bb0053-40b7-49df-9490-192e78a9b720","Type":"ContainerStarted","Data":"6efccdb7855156c04f2f7b2dff58f10b6436b26bad189db97a65342abc525e16"}
Oct 30 00:42:31.363076 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:31.363034    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-77hhk" event={"ID":"f69dcc85-0b97-43eb-8674-0ec1415c8dfb","Type":"ContainerStarted","Data":"8b819267f7bb1185abf55922a2797864493555e9eff9158feb54b2e7bed5057e"}
Oct 30 00:42:32.369646 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.369560    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wjf" event={"ID":"81da025c-1894-43c8-b83a-0b80409656ac","Type":"ContainerStarted","Data":"83e26191b6364d6f227113cc9f7d4a3bd68b0520348306812882e41575bd3c16"}
Oct 30 00:42:32.653106 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.653059    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-v9wjf" podStartSLOduration=400.690653692 podStartE2EDuration="6m44.6530421s" totalImagesPullingTime="3.962388408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:32.41580292 +0000 UTC m=+35.924309811" watchObservedRunningTime="2026-10-30 00:42:32.6530421 +0000 UTC m=+36.161548989"
Oct 30 00:42:32.653617 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.653591    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lmwj7"]
Oct 30 00:42:32.668836 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.668795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.678414 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.678379    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lmwj7"]
Oct 30 00:42:32.688317 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.688261    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:42:32.688317 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.688316    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:42:32.691624 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.688427    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b5pp6\""
Oct 30 00:42:32.750280 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.746344    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wmj68-h5vbk"]
Oct 30 00:42:32.765366 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.765332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:32.767364 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.767339    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wmj68-h5vbk"]
Oct 30 00:42:32.820778 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.820735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c849f30-20d0-4135-a2b2-e5d2daef76fa-crio-socket\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.820967 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.820791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.820967 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.820895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkbr\" (UniqueName: \"kubernetes.io/projected/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-api-access-whkbr\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.820967 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.820943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.821119 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.821053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c849f30-20d0-4135-a2b2-e5d2daef76fa-data-volume\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtt4\" (UniqueName: \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c849f30-20d0-4135-a2b2-e5d2daef76fa-crio-socket\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkbr\" (UniqueName: \"kubernetes.io/projected/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-api-access-whkbr\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.921970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.922062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c849f30-20d0-4135-a2b2-e5d2daef76fa-data-volume\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.922408    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c849f30-20d0-4135-a2b2-e5d2daef76fa-data-volume\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.922570    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c849f30-20d0-4135-a2b2-e5d2daef76fa-crio-socket\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:32.922641    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:32.922702    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:33.422681322 +0000 UTC m=+36.931188185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:32.923217    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:32.923329 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:32.923285    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:33.423268174 +0000 UTC m=+36.931775043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:32.948829 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:32.948768    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkbr\" (UniqueName: \"kubernetes.io/projected/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-api-access-whkbr\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:33.023429 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.023389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.023617 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.023447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.023617 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.023523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.023756 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.023704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtt4\" (UniqueName: \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.023999 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.023967    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.024113 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.023969    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.024402 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.024372    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.052959 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.052927    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtt4\" (UniqueName: \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.106347 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.106316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:33.377066 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.377029    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/0.log"
Oct 30 00:42:33.377646 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.377086    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9dac87d-7a94-43b5-ae69-38f6975388ed" containerID="0e70b52749d416b6ac792f9e5987059c4233324c073deea10e899e5f45e0b5ef" exitCode=255
Oct 30 00:42:33.377646 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.377156    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" event={"ID":"b9dac87d-7a94-43b5-ae69-38f6975388ed","Type":"ContainerDied","Data":"0e70b52749d416b6ac792f9e5987059c4233324c073deea10e899e5f45e0b5ef"}
Oct 30 00:42:33.377646 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.377490    2579 scope.go:122] "RemoveContainer" containerID="0e70b52749d416b6ac792f9e5987059c4233324c073deea10e899e5f45e0b5ef"
Oct 30 00:42:33.383165 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.383134    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r" event={"ID":"33045ca4-6bda-4fdc-9d44-f231e4161fae","Type":"ContainerStarted","Data":"65c04cacdad84bec094b15a101d13852ddcc6c650ea4946dc48b6405db444b19"}
Oct 30 00:42:33.429989 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.429230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:33.429989 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.429298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:33.429989 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:33.429473    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:33.429989 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:33.429538    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.4295198 +0000 UTC m=+37.938026677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:33.429989 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:33.429897    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:33.429989 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:33.429946    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:34.429930571 +0000 UTC m=+37.938437440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:33.433675 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.432982    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wmj68-h5vbk"]
Oct 30 00:42:33.462360 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:33.462206    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r" podStartSLOduration=400.256830088 podStartE2EDuration="6m45.462192773s" totalImagesPullingTime="5.205362685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:33.461748582 +0000 UTC m=+36.970255468" watchObservedRunningTime="2026-10-30 00:42:33.462192773 +0000 UTC m=+36.970699657"
Oct 30 00:42:34.314740 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.314693    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"]
Oct 30 00:42:34.338027 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.337995    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"]
Oct 30 00:42:34.338199 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.338077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"
Oct 30 00:42:34.357765 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.357734    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:42:34.357935 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.357870    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kbdjq\""
Oct 30 00:42:34.358008 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.357943    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:34.361917 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.361404    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"]
Oct 30 00:42:34.361917 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.361448    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"]
Oct 30 00:42:34.361917 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.361567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"
Oct 30 00:42:34.396035 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.395992    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26wcv" event={"ID":"59071108-f1ba-4346-beab-5cb766aecede","Type":"ContainerStarted","Data":"d22b6b6668b4454b1ea984db30d9f4c10e20bbe570a09a7f03c9c36f7365fdfc"}
Oct 30 00:42:34.398379 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.398344    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f" event={"ID":"db543397-1d27-4098-a794-7a57aaa7904b","Type":"ContainerStarted","Data":"0d9d525349eeef431f3d58cce30db1c666207165ee362614e598fdb220b9eaa5"}
Oct 30 00:42:34.400717 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.400292    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wmj68-h5vbk" event={"ID":"15ba866d-3fba-4bf9-beec-14ecda492c3f","Type":"ContainerStarted","Data":"dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7"}
Oct 30 00:42:34.400717 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.400334    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wmj68-h5vbk" event={"ID":"15ba866d-3fba-4bf9-beec-14ecda492c3f","Type":"ContainerStarted","Data":"6a7587e96d9aa2ca3dba26ed5500760dd5df7488d4cf688189d237032d408cb2"}
Oct 30 00:42:34.402321 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.402297    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/1.log"
Oct 30 00:42:34.402796 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.402778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/0.log"
Oct 30 00:42:34.402922 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.402816    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9dac87d-7a94-43b5-ae69-38f6975388ed" containerID="f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14" exitCode=255
Oct 30 00:42:34.402922 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.402880    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" event={"ID":"b9dac87d-7a94-43b5-ae69-38f6975388ed","Type":"ContainerDied","Data":"f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14"}
Oct 30 00:42:34.403041 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.402932    2579 scope.go:122] "RemoveContainer" containerID="0e70b52749d416b6ac792f9e5987059c4233324c073deea10e899e5f45e0b5ef"
Oct 30 00:42:34.403129 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.403109    2579 scope.go:122] "RemoveContainer" containerID="f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14"
Oct 30 00:42:34.403347 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.403323    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5phkz_openshift-console-operator(b9dac87d-7a94-43b5-ae69-38f6975388ed)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" podUID="b9dac87d-7a94-43b5-ae69-38f6975388ed"
Oct 30 00:42:34.441885 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.441852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:34.442050 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.441903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:34.442050 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.441987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpk5j\" (UniqueName: \"kubernetes.io/projected/d56b6c9e-b741-42c5-90d7-8c8c641064fa-kube-api-access-xpk5j\") pod \"migrator-66df5d496b-2kqhm\" (UID: \"d56b6c9e-b741-42c5-90d7-8c8c641064fa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"
Oct 30 00:42:34.442191 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.442154    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:34.442319 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.442223    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:36.442208441 +0000 UTC m=+39.950715326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:34.442319 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.442226    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:34.442319 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.442277    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:36.44226293 +0000 UTC m=+39.950769797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:34.510768 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.510716    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wmj68-h5vbk" podStartSLOduration=2.510697666 podStartE2EDuration="2.510697666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:34.510120569 +0000 UTC m=+38.018627448" watchObservedRunningTime="2026-10-30 00:42:34.510697666 +0000 UTC m=+38.019204574"
Oct 30 00:42:34.543525 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.543485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpk5j\" (UniqueName: \"kubernetes.io/projected/d56b6c9e-b741-42c5-90d7-8c8c641064fa-kube-api-access-xpk5j\") pod \"migrator-66df5d496b-2kqhm\" (UID: \"d56b6c9e-b741-42c5-90d7-8c8c641064fa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"
Oct 30 00:42:34.543936 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.543913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdscw\" (UniqueName: \"kubernetes.io/projected/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a-kube-api-access-bdscw\") pod \"migrator-66df5d496b-q74kq\" (UID: \"bb4302e4-1ec4-4ce1-a136-a91a5d1d306a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"
Oct 30 00:42:34.560084 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.560013    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f" podStartSLOduration=399.876810146 podStartE2EDuration="6m45.55999588s" totalImagesPullingTime="5.683185734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:34.551509622 +0000 UTC m=+38.060016514" watchObservedRunningTime="2026-10-30 00:42:34.55999588 +0000 UTC m=+38.068502768"
Oct 30 00:42:34.571987 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.571890    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpk5j\" (UniqueName: \"kubernetes.io/projected/d56b6c9e-b741-42c5-90d7-8c8c641064fa-kube-api-access-xpk5j\") pod \"migrator-66df5d496b-2kqhm\" (UID: \"d56b6c9e-b741-42c5-90d7-8c8c641064fa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"
Oct 30 00:42:34.610051 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.609753    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-26wcv" podStartSLOduration=2.927692613 podStartE2EDuration="8.6097314s" totalImagesPullingTime="5.682038787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:34.608808754 +0000 UTC m=+38.117315640" watchObservedRunningTime="2026-10-30 00:42:34.6097314 +0000 UTC m=+38.118238286"
Oct 30 00:42:34.645026 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.644984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdscw\" (UniqueName: \"kubernetes.io/projected/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a-kube-api-access-bdscw\") pod \"migrator-66df5d496b-q74kq\" (UID: \"bb4302e4-1ec4-4ce1-a136-a91a5d1d306a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"
Oct 30 00:42:34.665882 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.665852    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdscw\" (UniqueName: \"kubernetes.io/projected/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a-kube-api-access-bdscw\") pod \"migrator-66df5d496b-q74kq\" (UID: \"bb4302e4-1ec4-4ce1-a136-a91a5d1d306a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"
Oct 30 00:42:34.670662 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.670632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"
Oct 30 00:42:34.696962 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.696938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"
Oct 30 00:42:34.746320 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.746286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:34.746635 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.746613    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:34.746726 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.746687    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls podName:2deae516-c4e0-4377-98ee-084581c4b027 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:42.746669396 +0000 UTC m=+46.255176273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vfg9g" (UID: "2deae516-c4e0-4377-98ee-084581c4b027") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:34.847338 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.847243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:34.847981 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.847557    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:42:34.847981 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.847627    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls podName:0240e78b-bcf9-40a0-8707-0f7976d91005 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:42.847607186 +0000 UTC m=+46.356114062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n7f4l" (UID: "0240e78b-bcf9-40a0-8707-0f7976d91005") : secret "samples-operator-tls" not found
Oct 30 00:42:34.948232 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.948192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:34.948422 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.948258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:34.948422 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.948316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:34.948422 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.948356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:34.948422 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.948407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:34.948668 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948538    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:34.948668 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948554    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:34.948668 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:42.948590253 +0000 UTC m=+46.457097383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:34.948668 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948664    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:42.948654647 +0000 UTC m=+46.457161523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:34.948864 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948719    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:42:34.948864 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948745    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:42.948735501 +0000 UTC m=+46.457242379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : secret "router-metrics-certs-default" not found
Oct 30 00:42:34.948864 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948784    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:34.948864 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948791    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:34.948864 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948809    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:42.948803513 +0000 UTC m=+46.457310379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:34.948864 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948855    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:42:34.949154 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:34.948875    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert podName:85585df9-d495-4d44-bd82-398d44b76cf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:42.948869229 +0000 UTC m=+46.457376092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s248j" (UID: "85585df9-d495-4d44-bd82-398d44b76cf8") : secret "networking-console-plugin-cert" not found
Oct 30 00:42:34.987749 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:34.987713    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"]
Oct 30 00:42:34.992513 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:34.992446    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b6c9e_b741_42c5_90d7_8c8c641064fa.slice/crio-c3a8ab87de14c8248680e758b9da685aaf1ed8707618e9aab341875632a0f13a WatchSource:0}: Error finding container c3a8ab87de14c8248680e758b9da685aaf1ed8707618e9aab341875632a0f13a: Status 404 returned error can't find the container with id c3a8ab87de14c8248680e758b9da685aaf1ed8707618e9aab341875632a0f13a
Oct 30 00:42:35.035839 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.035806    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"]
Oct 30 00:42:35.037969 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:35.037929    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4302e4_1ec4_4ce1_a136_a91a5d1d306a.slice/crio-ff41799c38e73f9a43ffc3e6b69a89cbf51f02d1c7b58331dde4a30d502054fe WatchSource:0}: Error finding container ff41799c38e73f9a43ffc3e6b69a89cbf51f02d1c7b58331dde4a30d502054fe: Status 404 returned error can't find the container with id ff41799c38e73f9a43ffc3e6b69a89cbf51f02d1c7b58331dde4a30d502054fe
Oct 30 00:42:35.048982 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.048924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:35.049133 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:35.049117    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:42:35.049186 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:35.049181    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls podName:49fcf677-f503-492a-90e8-c95e329ee6f6 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:43.049166577 +0000 UTC m=+46.557673452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls") pod "dns-default-dkpn9" (UID: "49fcf677-f503-492a-90e8-c95e329ee6f6") : secret "dns-default-metrics-tls" not found
Oct 30 00:42:35.410480 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.409977    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5" event={"ID":"56ea1b39-0a84-4b31-a200-0c1b68cfc927","Type":"ContainerStarted","Data":"ad19f4f1953b96abb32fc2fecb73f1b54d46a6e010c21d1c8a47abec0ae969a5"}
Oct 30 00:42:35.411817 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.411785    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/1.log"
Oct 30 00:42:35.412665 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.412115    2579 scope.go:122] "RemoveContainer" containerID="f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14"
Oct 30 00:42:35.412665 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:35.412325    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5phkz_openshift-console-operator(b9dac87d-7a94-43b5-ae69-38f6975388ed)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" podUID="b9dac87d-7a94-43b5-ae69-38f6975388ed"
Oct 30 00:42:35.413131 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.413106    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq" event={"ID":"bb4302e4-1ec4-4ce1-a136-a91a5d1d306a","Type":"ContainerStarted","Data":"ff41799c38e73f9a43ffc3e6b69a89cbf51f02d1c7b58331dde4a30d502054fe"}
Oct 30 00:42:35.414513 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.414490    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm" event={"ID":"d56b6c9e-b741-42c5-90d7-8c8c641064fa","Type":"ContainerStarted","Data":"c3a8ab87de14c8248680e758b9da685aaf1ed8707618e9aab341875632a0f13a"}
Oct 30 00:42:35.454467 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.454391    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5" podStartSLOduration=397.702202937 podStartE2EDuration="6m44.454377574s" totalImagesPullingTime="6.752174637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:35.452560346 +0000 UTC m=+38.961067233" watchObservedRunningTime="2026-10-30 00:42:35.454377574 +0000 UTC m=+38.962884458"
Oct 30 00:42:35.465165 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.465134    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-q48h7"]
Oct 30 00:42:35.485981 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.485950    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-q48h7"]
Oct 30 00:42:35.486145 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.486027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.502613 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.502583    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 00:42:35.502773 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.502592    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 00:42:35.502773 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.502701    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 00:42:35.502902 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.502869    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fmf46\""
Oct 30 00:42:35.503012 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.502964    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 00:42:35.679517 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.676572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-key\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.679517 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.676720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-cabundle\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.679517 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.676801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2snrg\" (UniqueName: \"kubernetes.io/projected/e0532776-f74d-4d8a-832a-6719d3fec3c2-kube-api-access-2snrg\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.778325 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.778280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-cabundle\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.778515 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.778374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2snrg\" (UniqueName: \"kubernetes.io/projected/e0532776-f74d-4d8a-832a-6719d3fec3c2-kube-api-access-2snrg\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.778515 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.778465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-key\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.779584 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.779556    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-cabundle\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.781869 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.781846    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-key\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.804497 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.804445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2snrg\" (UniqueName: \"kubernetes.io/projected/e0532776-f74d-4d8a-832a-6719d3fec3c2-kube-api-access-2snrg\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:35.812299 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:35.812101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:36.188626 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.188590    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-q48h7"]
Oct 30 00:42:36.190030 ip-10-0-130-169 kubenswrapper[2579]: W1030 00:42:36.189997    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0532776_f74d_4d8a_832a_6719d3fec3c2.slice/crio-c304b3252841b1a3e7bd31eaefd05a3a89bd8c370d1d2af68fd19551e7d71ad4 WatchSource:0}: Error finding container c304b3252841b1a3e7bd31eaefd05a3a89bd8c370d1d2af68fd19551e7d71ad4: Status 404 returned error can't find the container with id c304b3252841b1a3e7bd31eaefd05a3a89bd8c370d1d2af68fd19551e7d71ad4
Oct 30 00:42:36.421128 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.421076    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7" event={"ID":"e0532776-f74d-4d8a-832a-6719d3fec3c2","Type":"ContainerStarted","Data":"b78a6c394541b78e3384b9a83e616cabd7ad84f0bab9c99e0b39dd20477cd420"}
Oct 30 00:42:36.421607 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.421155    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7" event={"ID":"e0532776-f74d-4d8a-832a-6719d3fec3c2","Type":"ContainerStarted","Data":"c304b3252841b1a3e7bd31eaefd05a3a89bd8c370d1d2af68fd19551e7d71ad4"}
Oct 30 00:42:36.423090 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.423063    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-77hhk" event={"ID":"f69dcc85-0b97-43eb-8674-0ec1415c8dfb","Type":"ContainerStarted","Data":"0b1c78009a89f252f33ac6ff689d31bd84743285cf8dcbdbb5e6ec2ecbda9956"}
Oct 30 00:42:36.463870 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.463774    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7" podStartSLOduration=1.463759384 podStartE2EDuration="1.463759384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:36.463213232 +0000 UTC m=+39.971720117" watchObservedRunningTime="2026-10-30 00:42:36.463759384 +0000 UTC m=+39.972266268"
Oct 30 00:42:36.487575 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.487536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:36.487752 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.487597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:36.487900 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:36.487885    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:36.487950 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:36.487944    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.487926963 +0000 UTC m=+43.996433832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:36.488534 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:36.488493    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:36.488666 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:36.488563    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.488543724 +0000 UTC m=+43.997050610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:36.507889 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:36.507834    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-77hhk" podStartSLOduration=34.11562603 podStartE2EDuration="39.507812674s" totalImagesPullingTime="5.392186644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:36.506270482 +0000 UTC m=+40.014777378" watchObservedRunningTime="2026-10-30 00:42:36.507812674 +0000 UTC m=+40.016319564"
Oct 30 00:42:37.231322 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.231287    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rxj86_7d027727-5c89-44a2-b1b8-20661fd13909/dns-node-resolver/0.log"
Oct 30 00:42:37.410473 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.410007    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:37.410473 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.410084    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:37.410693 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.410525    2579 scope.go:122] "RemoveContainer" containerID="f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14"
Oct 30 00:42:37.420474 ip-10-0-130-169 kubenswrapper[2579]: E1030 00:42:37.410752    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5phkz_openshift-console-operator(b9dac87d-7a94-43b5-ae69-38f6975388ed)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" podUID="b9dac87d-7a94-43b5-ae69-38f6975388ed"
Oct 30 00:42:37.428760 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.428721    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq" event={"ID":"bb4302e4-1ec4-4ce1-a136-a91a5d1d306a","Type":"ContainerStarted","Data":"b124c35f8105034af619c8e17f529852af40b08ef405db67a9f9d272f2a9ad17"}
Oct 30 00:42:37.431255 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.431222    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm" event={"ID":"d56b6c9e-b741-42c5-90d7-8c8c641064fa","Type":"ContainerStarted","Data":"1bc35fb0a669d89ee39b25978a41a7de56edd80728b5f94ed1242c2c877d0702"}
Oct 30 00:42:37.431798 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.431737    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:37.517554 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.517141    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq" podStartSLOduration=1.656192959 podStartE2EDuration="3.517121357s" totalImagesPullingTime="1.860928398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:37.50201138 +0000 UTC m=+41.010518265" watchObservedRunningTime="2026-10-30 00:42:37.517121357 +0000 UTC m=+41.025628243"
Oct 30 00:42:37.547494 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:37.545480    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm" podStartSLOduration=1.636663046 podStartE2EDuration="3.545446352s" totalImagesPullingTime="1.908783306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:37.544845806 +0000 UTC m=+41.053352693" watchObservedRunningTime="2026-10-30 00:42:37.545446352 +0000 UTC m=+41.053953239"
Oct 30 00:42:38.232084 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:38.232056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r6gln_e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6/node-ca/0.log"
Oct 30 00:42:38.386486 ip-10-0-130-169 kubenswrapper[2579]: I1030 00:42:38.386252    2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:42:38.386400 ip-10-0-130-169 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:42:38.398815 ip-10-0-130-169 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:42:38.398989 ip-10-0-130-169 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:42:38.399042 ip-10-0-130-169 systemd[1]: kubelet.service: Consumed 3.349s CPU time.
Oct 30 00:42:38.410151 ip-10-0-130-169 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:42:38.481571 ip-10-0-130-169 kubenswrapper[5104]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:42:38.481571 ip-10-0-130-169 kubenswrapper[5104]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:42:38.481571 ip-10-0-130-169 kubenswrapper[5104]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:42:38.481571 ip-10-0-130-169 kubenswrapper[5104]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:42:38.483819 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483732    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.483819 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483818    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.483819 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483823    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483826    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483829    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483832    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483835    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483838    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483841    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483844    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483846    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483849    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483852    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483855    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483857    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483860    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483863    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483865    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483868    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483871    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483878    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483881    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.483917 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483883    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483886    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483889    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483892    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483894    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483897    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483900    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483903    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483905    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483908    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483910    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483913    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483916    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483918    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483921    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483924    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483927    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483930    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483933    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.484404 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483935    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483938    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483940    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483943    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483946    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483948    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483951    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483953    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483956    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483959    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483961    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483964    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483967    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483970    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483973    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483975    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483978    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483981    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483984    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.484930 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483987    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483989    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483992    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.483996    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484000    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484003    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484006    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484009    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484012    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484015    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484017    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484020    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484026    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484029    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484032    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484034    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484037    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484039    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484042    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.485384 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484045    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484050    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484054    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484057    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484061    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484064    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484067    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484069    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484072    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484075    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484079    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484081    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484084    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484087    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484089    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484092    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484094    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484097    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484100    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.485850 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484102    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484105    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484109    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484112    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484115    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484117    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484120    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484123    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484933    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484942    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484945    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484948    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484951    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484954    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484957    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484960    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484962    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484965    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484968    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484970    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.486317 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484973    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484975    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484978    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484981    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484984    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484986    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484989    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484992    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484994    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484997    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.484999    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485002    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485004    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485008    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485011    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485013    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485015    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485018    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485020    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485023    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.486838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485026    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485029    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485031    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485034    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485036    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485039    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485042    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485044    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485047    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485049    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485052    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485054    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485057    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485060    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485063    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485066    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485069    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485071    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485074    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485076    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.487351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485079    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485081    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485084    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485087    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485089    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485092    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485095    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485097    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485100    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485102    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485105    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485107    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485110    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485113    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485115    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485118    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485120    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485123    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485125    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485128    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.487859 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485130    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485134    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485136    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485139    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485141    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485144    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485146    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485149    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485151    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485154    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485156    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485159    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485162    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485168    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485171    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485175    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485178    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485182    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485185    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.488331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485188    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485191    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485194    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485197    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485200    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485202    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485205    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485208    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485211    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485214    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485219    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485223    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485226    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485229    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485231    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485708    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485713    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485716    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485719    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.488815 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485722    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485725    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485727    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485730    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485733    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485736    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485739    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485741    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485744    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485747    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485749    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485752    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485755    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485757    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485760    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485762    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485766    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485768    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485771    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485773    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.489310 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485776    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485779    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485782    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485784    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485787    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485790    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485792    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485795    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485797    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485800    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485802    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485805    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485807    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485810    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485813    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485816    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485818    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485821    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485824    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.489835 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485826    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485829    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485831    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485834    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485837    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485840    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485842    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485845    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485847    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485850    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485853    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485856    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485858    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485861    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485864    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485867    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485870    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485873    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485875    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485878    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.490327 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485880    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485883    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485885    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485887    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485890    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485892    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485895    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485901    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485904    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485907    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485910    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485913    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485917    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485920    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485922    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485925    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485927    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485930    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485932    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.490820 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485935    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485937    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485940    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485942    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485947    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485951    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485954    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485956    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485959    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485962    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485965    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485968    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485970    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485972    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485975    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485978    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485983    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485986    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485988    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.491373 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485991    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485994    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.485997    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486000    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486002    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486093    5104 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486100    5104 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486106    5104 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486111    5104 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486116    5104 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486119    5104 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486125    5104 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486130    5104 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486133    5104 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486136    5104 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486140    5104 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486143    5104 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486146    5104 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486149    5104 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486152    5104 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486155    5104 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486158    5104 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486161    5104 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486166    5104 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:42:38.492151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486169    5104 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486172    5104 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486175    5104 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486179    5104 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486183    5104 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486187    5104 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486190    5104 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486194    5104 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486197    5104 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486202    5104 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486205    5104 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486208    5104 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486211    5104 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486215    5104 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486218    5104 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486221    5104 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486224    5104 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486227    5104 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486230    5104 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486234    5104 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486237    5104 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486240    5104 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486243    5104 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486246    5104 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:42:38.493056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486250    5104 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486253    5104 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486256    5104 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486259    5104 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486263    5104 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486265    5104 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486268    5104 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486273    5104 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486276    5104 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486279    5104 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486282    5104 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486286    5104 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486289    5104 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486294    5104 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486297    5104 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486300    5104 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486303    5104 flags.go:64] FLAG: --help="false"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486308    5104 flags.go:64] FLAG: --hostname-override="ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486311    5104 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486314    5104 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486317    5104 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486320    5104 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486324    5104 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486327    5104 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:42:38.493682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486330    5104 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486333    5104 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486336    5104 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486339    5104 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486342    5104 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486345    5104 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486348    5104 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486351    5104 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486354    5104 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486357    5104 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486360    5104 flags.go:64] FLAG: --lock-file=""
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486363    5104 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486366    5104 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486369    5104 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486375    5104 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486378    5104 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486382    5104 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486385    5104 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486388    5104 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486391    5104 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486394    5104 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486398    5104 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486402    5104 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486406    5104 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486410    5104 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:42:38.494267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486414    5104 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486419    5104 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486424    5104 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486428    5104 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486433    5104 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486438    5104 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486442    5104 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486464    5104 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486468    5104 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486471    5104 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486474    5104 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486477    5104 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486480    5104 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486483    5104 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486486    5104 flags.go:64] FLAG: --port="10250"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486489    5104 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486492    5104 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0377050885885c137"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486496    5104 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486499    5104 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486502    5104 flags.go:64] FLAG: --register-node="true"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486504    5104 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486509    5104 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486511    5104 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486514    5104 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486518    5104 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486522    5104 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:42:38.494882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486525    5104 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486528    5104 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486533    5104 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486536    5104 flags.go:64] FLAG: --runonce="false"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486539    5104 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486542    5104 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486546    5104 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486549    5104 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486552    5104 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486557    5104 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486560    5104 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486563    5104 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486566    5104 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486569    5104 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486572    5104 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486575    5104 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486578    5104 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486581    5104 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486584    5104 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486589    5104 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486592    5104 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486595    5104 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486600    5104 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486603    5104 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486605    5104 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:42:38.495532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486608    5104 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486611    5104 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486614    5104 flags.go:64] FLAG: --v="2"
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486618    5104 flags.go:64] FLAG: --version="false"
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486623    5104 flags.go:64] FLAG: --vmodule=""
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486627    5104 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.486631    5104 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486760    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486763    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486768    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486771    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486773    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486776    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486780    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486782    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486785    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486788    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486790    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486793    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486795    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486798    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.496244 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486801    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486805    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486808    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486812    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486814    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486817    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486820    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486822    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486827    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486831    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486834    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486837    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486840    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486842    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486845    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486848    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486851    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486854    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486857    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.496782 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486860    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486864    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486866    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486869    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486872    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486875    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486879    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486881    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486884    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486886    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486889    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486891    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486894    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486897    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486899    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486902    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486906    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486909    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486911    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.497281 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486914    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486917    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486919    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486921    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486924    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486927    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486929    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486932    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486934    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486937    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486939    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486942    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486944    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486947    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486952    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486955    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486958    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486961    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486964    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486967    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.497874 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486969    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486972    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486975    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486977    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486980    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486982    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486985    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486988    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486991    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486993    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486996    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.486999    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487001    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487003    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487006    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487009    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487012    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487014    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487017    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487019    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.498439 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487022    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487024    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487027    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487029    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487032    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487034    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487038    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487041    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487044    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487046    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487051    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487053    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487056    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487059    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.487084    5104 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487118    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.498987 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487122    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487125    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487128    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487131    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487133    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487136    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487139    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487142    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487144    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487147    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487149    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487152    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487154    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487157    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487160    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487162    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487165    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487167    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487170    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487172    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.499611 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487175    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487178    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487180    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487185    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487187    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487190    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487194    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487197    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487200    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487202    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487205    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487207    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487210    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487213    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487215    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487218    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487221    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487223    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487226    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.500540 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487228    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487231    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487233    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487236    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487238    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487241    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487243    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487246    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487248    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487251    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487253    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487256    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487258    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487261    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487263    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487266    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487270    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487273    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487277    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.501233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487281    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487284    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487286    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487289    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487291    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487294    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487297    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487299    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487302    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487304    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487308    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487310    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487313    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487315    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487318    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487320    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487323    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487327    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487331    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.501849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487334    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487336    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487339    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487342    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487344    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487348    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487351    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487355    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487358    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487361    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487364    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487367    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487370    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487374    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487377    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487380    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487382    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487385    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487388    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.502357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487390    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487393    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487395    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487399    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487401    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487404    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487406    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487409    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.487411    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.492933    5104 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.492951    5104 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.492981    5104 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.492987    5104 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493046    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493052    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493056    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493058    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493061    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493064    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.502868 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493067    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493070    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493073    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493075    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493078    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493081    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493083    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493086    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493089    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493093    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493095    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493098    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493101    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493104    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493106    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493109    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493112    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493114    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493117    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493119    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.503335 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493122    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493125    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493128    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493131    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493135    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493140    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493143    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493146    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493149    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493151    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493154    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493156    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493159    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493161    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493164    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493167    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493170    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493173    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493175    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.503838 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493178    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493181    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493184    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493186    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493189    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493192    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493195    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493197    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493200    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493202    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493205    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493207    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493210    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493212    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493215    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493218    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493221    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493223    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493226    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.504357 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493228    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493231    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493233    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493236    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493239    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493241    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493244    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493247    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493249    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493252    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493255    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493257    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493261    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493265    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493268    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493272    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493275    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493277    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493280    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.504839 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493282    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493285    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493287    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493290    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493293    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493295    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493298    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493301    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493303    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493306    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493308    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493311    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493313    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493316    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493319    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493321    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493324    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493327    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493329    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493332    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.505295 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493335    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493338    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493340    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.493369    5104 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493561    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493567    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493571    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493574    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493578    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493582    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493585    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493588    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493591    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493594    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493596    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493599    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.505914 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493601    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493604    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493607    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493609    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493612    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493615    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493617    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493620    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493623    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493625    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493628    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493631    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493633    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493636    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493639    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493641    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493644    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493647    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493649    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493652    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.506292 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493655    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493657    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493660    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493663    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493665    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493668    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493672    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493675    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493678    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493680    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493683    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493685    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493688    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493691    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493693    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493696    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493699    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493702    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493705    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493708    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.506893 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493711    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493713    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493716    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493718    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493721    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493724    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493726    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493729    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493731    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493734    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493738    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493741    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493744    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493746    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493749    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493752    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493755    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493757    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493760    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.507387 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493765    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493769    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493771    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493774    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493777    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493779    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493782    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493784    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493787    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493789    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493792    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493794    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493797    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493800    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493803    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493805    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493808    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493811    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493813    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.507862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493816    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493818    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493821    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493823    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493826    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493828    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493831    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493834    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493836    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493842    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493844    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493858    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493862    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493865    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493868    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493871    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.508363 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.493899    5104 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493933    5104 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493937    5104 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493940    5104 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493942    5104 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493945    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493948    5104 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493950    5104 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493953    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493956    5104 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493959    5104 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493961    5104 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493964    5104 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493967    5104 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493970    5104 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493972    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493977    5104 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493979    5104 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493982    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493985    5104 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:42:38.508775 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493987    5104 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493990    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493993    5104 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493995    5104 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.493998    5104 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494001    5104 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494006    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494008    5104 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494011    5104 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494014    5104 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494016    5104 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494019    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494022    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494025    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494027    5104 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494030    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494033    5104 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494035    5104 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494038    5104 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494041    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:42:38.509365 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494043    5104 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494046    5104 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494048    5104 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494051    5104 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494054    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494057    5104 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494059    5104 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494062    5104 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494066    5104 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494069    5104 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494071    5104 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494074    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494076    5104 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494079    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494081    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494084    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494087    5104 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494089    5104 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494093    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494096    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:42:38.510331 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494099    5104 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494101    5104 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494104    5104 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494106    5104 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494109    5104 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494112    5104 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494114    5104 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494119    5104 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494129    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494133    5104 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494136    5104 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494139    5104 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494141    5104 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494144    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494147    5104 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494149    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494153    5104 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494157    5104 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494160    5104 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:42:38.511138 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494162    5104 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494167    5104 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494169    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494172    5104 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494175    5104 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494177    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494180    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494183    5104 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494185    5104 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494188    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494191    5104 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494194    5104 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494197    5104 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494200    5104 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494202    5104 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494205    5104 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494208    5104 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494210    5104 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494213    5104 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:42:38.511942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494216    5104 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494219    5104 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494221    5104 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494224    5104 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494227    5104 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494229    5104 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494233    5104 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494236    5104 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:38.494239    5104 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.494296    5104 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.495310    5104 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.495363    5104 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.495551    5104 server.go:1014] "Starting client certificate rotation"
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.495697    5104 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.495756    5104 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:36:56 +0000 UTC" deadline="2028-08-11 00:55:19.923968838 +0000 UTC"
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.495792    5104 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15624h12m41.428179677s"
Oct 30 00:42:38.512834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.496133    5104 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:42:38.513719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.497247    5104 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:42:38.513719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.498745    5104 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:42:38.513719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.501053    5104 log.go:25] "Validated CRI v1 image API"
Oct 30 00:42:38.513719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.501877    5104 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:42:38.513719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.503909    5104 fs.go:135] Filesystem UUIDs: map[2adee509-3055-4353-8f3d-e2b44b78f014:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c9e86c4c-d3e5-43b6-9f5a-19788d739546:/dev/nvme0n1p4]
Oct 30 00:42:38.513999 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.503935    5104 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/303442b5c031dfa67e64696ad482f1f3466163bbf51b19d452685f0a7663048e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/303442b5c031dfa67e64696ad482f1f3466163bbf51b19d452685f0a7663048e/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a4e1b78a880e1c90c59e2af2ac50f73b75de77b0ad0cd58c817ac5dfa706d75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a4e1b78a880e1c90c59e2af2ac50f73b75de77b0ad0cd58c817ac5dfa706d75/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/501238e2345bfe4f9ebfa0008d8f4b12e0993b7905f8f4bad3f7f3393b72ccc7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/501238e2345bfe4f9ebfa0008d8f4b12e0993b7905f8f4bad3f7f3393b72ccc7/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/532cd48be8aa6b55ce672db105d7bf6c304417708938946b2921e4ca89175d68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/532cd48be8aa6b55ce672db105d7bf6c304417708938946b2921e4ca89175d68/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/586e2881c8d32bdea15368b7446c8b0d8724e12051d75c475edefbf039d431b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/586e2881c8d32bdea15368b7446c8b0d8724e12051d75c475edefbf039d431b3/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/615b46a4c54925eacc7cbf15bec37c7ad0c000436b828c0301bbab676933ecca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/615b46a4c54925eacc7cbf15bec37c7ad0c000436b828c0301bbab676933ecca/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6182d2c6e3c810fa7d0495e0e45b24f1383e7042c185a72d40390c9f5e88668e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6182d2c6e3c810fa7d0495e0e45b24f1383e7042c185a72d40390c9f5e88668e/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64e2b2035554c6d65d1cb62e61ebe849dbc427fd3a978173782c3a3df331a3d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64e2b2035554c6d65d1cb62e61ebe849dbc427fd3a978173782c3a3df331a3d1/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a7587e96d9aa2ca3dba26ed5500760dd5df7488d4cf688189d237032d408cb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a7587e96d9aa2ca3dba26ed5500760dd5df7488d4cf688189d237032d408cb2/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cb4d9d883799c0de98745bc1149013e88d45d035c0c616121bb0b60ea1c0d23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cb4d9d883799c0de98745bc1149013e88d45d035c0c616121bb0b60ea1c0d23/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6efccdb7855156c04f2f7b2dff58f10b6436b26bad189db97a65342abc525e16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6efccdb7855156c04f2f7b2dff58f10b6436b26bad189db97a65342abc525e16/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7484c3c21e65844f4ea8b48d76affa6dc60ed06e922e34be59b4f17c2b124071/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7484c3c21e65844f4ea8b48d76affa6dc60ed06e922e34be59b4f17c2b124071/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b819267f7bb1185abf55922a2797864493555e9eff9158feb54b2e7bed5057e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b819267f7bb1185abf55922a2797864493555e9eff9158feb54b2e7bed5057e/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d1037f0ef4eb537f025b0e04f1a7a749ee07d715e4ff5880c66056112dbc727/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d1037f0ef4eb537f025b0e04f1a7a749ee07d715e4ff5880c66056112dbc727/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9063c68f92d2c33f53d770a9dedc6c9e0372507b62d00dda32a636db5999e5aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9063c68f92d2c33f53d770a9dedc6c9e0372507b62d00dda32a636db5999e5aa/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96fae5a4031880a78bac9a755c06367fa3a998e276d66611b25f4bb6dcc7e06c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96fae5a4031880a78bac9a755c06367fa3a998e276d66611b25f4bb6dcc7e06c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdbb2c8206606e39d673e1c3da53ada1679b600e07a2235916f856cfc32041b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdbb2c8206606e39d673e1c3da53ada1679b600e07a2235916f856cfc32041b6/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c304b3252841b1a3e7bd31eaefd05a3a89bd8c370d1d2af68fd19551e7d71ad4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c304b3252841b1a3e7bd31eaefd05a3a89bd8c370d1d2af68fd19551e7d71ad4/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3a8ab87de14c8248680e758b9da685aaf1ed8707618e9aab341875632a0f13a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3a8ab87de14c8248680e758b9da685aaf1ed8707618e9aab341875632a0f13a/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccc5ef2e03bbc6dbb03dc349ae32c22dcb1a465242ba95c8786edd7f44f44ac3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccc5ef2e03bbc6dbb03dc349ae32c22dcb1a465242ba95c8786edd7f44f44ac3/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d50536e56bdf06f88016b7a66ac227f253b006e76d13557570c101b66efca658/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d50536e56bdf06f88016b7a66ac227f253b006e76d13557570c101b66efca658/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8fa684a7e915bc9a5737948f2a4f9087640d6ef8ecc4db2b47580327fba7095/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8fa684a7e915bc9a5737948f2a4f9087640d6ef8ecc4db2b47580327fba7095/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebee0ab5c207b2661616886648299b32db2db9f14b4ce0f50312e859067fcc96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebee0ab5c207b2661616886648299b32db2db9f14b4ce0f50312e859067fcc96/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff41799c38e73f9a43ffc3e6b69a89cbf51f02d1c7b58331dde4a30d502054fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff41799c38e73f9a43ffc3e6b69a89cbf51f02d1c7b58331dde4a30d502054fe/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0240e78b-bcf9-40a0-8707-0f7976d91005/volumes/kubernetes.io~projected/kube-api-access-gql2j:{mountpoint:/var/lib/kubelet/pods/0240e78b-bcf9-40a0-8707-0f7976d91005/volumes/kubernetes.io~projected/kube-api-access-gql2j major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volumes/kubernetes.io~projected/kube-api-access-p62hj:{mountpoint:/var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volumes/kubernetes.io~projected/kube-api-access-p62hj major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15ba866d-3fba-4bf9-beec-14ecda492c3f/volumes/kubernetes.io~projected/kube-api-access-pxtt4:{mountpoint:/var/lib/kubelet/pods/15ba866d-3fba-4bf9-beec-14ecda492c3f/volumes/kubernetes.io~projected/kube-api-access-pxtt4 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c849f30-20d0-4135-a2b2-e5d2daef76fa/volumes/kubernetes.io~projected/kube-api-access-whkbr:{mountpoint:/var/lib/kubelet/pods/1c849f30-20d0-4135-a2b2-e5d2daef76fa/volumes/kubernetes.io~projected/kube-api-access-whkbr major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~projected/kube-api-access-rlj97:{mountpoint:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~projected/kube-api-access-rlj97 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2deae516-c4e0-4377-98ee-084581c4b027/volumes/kubernetes.io~projected/kube-api-access-5pw8p:{mountpoint:/var/lib/kubelet/pods/2deae516-c4e0-4377-98ee-084581c4b027/volumes/kubernetes.io~projected/kube-api-access-5pw8p major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405/volumes/kubernetes.io~projected/kube-api-access-4cnf6:{mountpoint:/var/lib/kubelet/pods/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405/volumes/kubernetes.io~projected/kube-api-access-4cnf6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33045ca4-6bda-4fdc-9d44-f231e4161fae/volumes/kubernetes.io~projected/kube-api-access-sl9qt:{mountpoint:/var/lib/kubelet/pods/33045ca4-6bda-4fdc-9d44-f231e4161fae/volumes/kubernetes.io~projected/kube-api-access-sl9qt major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33045ca4-6bda-4fdc-9d44-f231e4161fae/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/33045ca4-6bda-4fdc-9d44-f231e4161fae/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254ef03-935f-47d4-948a-70855cb3afb8/volumes/kubernetes.io~projected/kube-api-access-4l9nl:{mountpoint:/var/lib/kubelet/pods/4254ef03-935f-47d4-948a-70855cb3afb8/volumes/kubernetes.io~projected/kube-api-access-4l9nl major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~projected/kube-api-access-fzt92:{mountpoint:/var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~projected/kube-api-access-fzt92 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~secret/default-certificate major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~secret/stats-auth major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49fcf677-f503-492a-90e8-c95e329ee6f6/volumes/kubernetes.io~projected/kube-api-access-wp67x:{mountpoint:/var/lib/kubelet/pods/49fcf677-f503-492a-90e8-c95e329ee6f6/volumes/kubernetes.io~projected/kube-api-access-wp67x major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~projected/kube-api-access-6sfzl:{mountpoint:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~projected/kube-api-access-6sfzl major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56ea1b39-0a84-4b31-a200-0c1b68cfc927/volumes/kubernetes.io~projected/kube-api-access-w7qq9:{mountpoint:/var/lib/kubelet/pods/56ea1b39-0a84-4b31-a200-0c1b68cfc927/volumes/kubernetes.io~projected/kube-api-access-w7qq9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59071108-f1ba-4346-beab-5cb766aecede/volumes/kubernetes.io~projected/kube-api-access-29q9f:{mountpoint:/var/lib/kubelet/pods/59071108-f1ba-4346-beab-5cb766aecede/volumes/kubernetes.io~projected/kube-api-access-29q9f major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59071108-f1ba-4346-beab-5cb766aecede/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/59071108-f1ba-4346-beab-5cb766aecede/volumes/kubernetes.io~secret/cert major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a4543fe-837e-4480-b419-974b723e4475/volumes/kubernetes.io~projected/kube-api-access-fd285:{mountpoint:/var/lib/kubelet/pods/6a4543fe-837e-4480-b419-974b723e4475/volumes/kubernetes.io~projected/kube-api-access-fd285 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7469affe-9d00-44b3-a833-4ca25b86da5c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7469affe-9d00-44b3-a833-4ca25b86da5c/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79ef76c0-8e88-4b4a-8d6a-83da18b39d78/volumes/kubernetes.io~projected/kube-api-access-55ncq:{mountpoint:/var/lib/kubelet/pods/79ef76c0-8e88-4b4a-8d6a-83da18b39d78/volumes/kubernetes.io~projected/kube-api-access-55ncq major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d027727-5c89-44a2-b1b8-20661fd13909/volumes/kubernetes.io~projected/kube-api-access-2r7cp:{mountpoint:/var/lib/kubelet/pods/7d027727-5c89-44a2-b1b8-20661fd13909/volumes/kubernetes.io~projected/kube-api-access-2r7cp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81da025c-1894-43c8-b83a-0b80409656ac/volumes/kubernetes.io~projected/kube-api-access-5dt8t:{mountpoint:/var/lib/kubelet/pods/81da025c-1894-43c8-b83a-0b80409656ac/volumes/kubernetes.io~projected/kube-api-access-5dt8t major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81da025c-1894-43c8-b83a-0b80409656ac/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/81da025c-1894-43c8-b83a-0b80409656ac/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/927717eb-ca5c-4f65-8a7a-bdc83054080c/volumes/kubernetes.io~projected/kube-api-access-l6b25:{mountpoint:/var/lib/kubelet/pods/927717eb-ca5c-4f65-8a7a-bdc83054080c/volumes/kubernetes.io~projected/kube-api-access-l6b25 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cec9dfe-c016-4228-be90-4724120f7260/volumes/kubernetes.io~projected/kube-api-access-z546g:{mountpoint:/var/lib/kubelet/pods/9cec9dfe-c016-4228-be90-4724120f7260/volumes/kubernetes.io~projected/kube-api-access-z546g major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8d8e7da-34ee-466e-bf3d-48a8a56b722f/volumes/kubernetes.io~projected/kube-api-access-vdgzs:{mountpoint:/var/lib/kubelet/pods/a8d8e7da-34ee-466e-bf3d-48a8a56b722f/volumes/kubernetes.io~projected/kube-api-access-vdgzs major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e082e1-ce44-4deb-8036-89d2ae8dcc46/volumes/kubernetes.io~projected/kube-api-access-p7j55:{mountpoint:/var/lib/kubelet/pods/a8e082e1-ce44-4deb-8036-89d2ae8dcc46/volumes/kubernetes.io~projected/kube-api-access-p7j55 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3bb0053-40b7-49df-9490-192e78a9b720/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b3bb0053-40b7-49df-9490-192e78a9b720/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9dac87d-7a94-43b5-ae69-38f6975388ed/volumes/kubernetes.io~projected/kube-api-access-d5tx8:{mountpoint:/var/lib/kubelet/pods/b9dac87d-7a94-43b5-ae69-38f6975388ed/volumes/kubernetes.io~projected/kube-api-access-d5tx8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9dac87d-7a94-43b5-ae69-38f6975388ed/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b9dac87d-7a94-43b5-ae69-38f6975388ed/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a/volumes/kubernetes.io~projected/kube-api-access-bdscw:{mountpoint:/var/lib/kubelet/pods/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a/volumes/kubernetes.io~projected/kube-api-access-bdscw major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce14d2cc-a9e8-49af-be37-9b8e6a433c65/volumes/kubernetes.io~projected/kube-api-access-cg87z:{mountpoint:/var/lib/kubelet/pods/ce14d2cc-a9e8-49af-be37-9b8e6a433c65/volumes/kubernetes.io~projected/kube-api-access-cg87z major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d56b6c9e-b741-42c5-90d7-8c8c641064fa/volumes/kubernetes.io~projected/kube-api-access-xpk5j:{mountpoint:/var/lib/kubelet/pods/d56b6c9e-b741-42c5-90d7-8c8c641064fa/volumes/kubernetes.io~projected/kube-api-access-xpk5j major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~projected/kube-api-access-qrgrf:{mountpoint:/var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~projected/kube-api-access-qrgrf major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db543397-1d27-4098-a794-7a57aaa7904b/volumes/kubernetes.io~projected/kube-api-access-ctjfv:{mountpoint:/var/lib/kubelet/pods/db543397-1d27-4098-a794-7a57aaa7904b/volumes/kubernetes.io~projected/kube-api-access-ctjfv major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db543397-1d27-4098-a794-7a57aaa7904b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/db543397-1d27-4098-a794-7a57aaa7904b/volumes/kubernetes.io~secret/serving-cert major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0532776-f74d-4d8a-832a-6719d3fec3c2/volumes/kubernetes.io~projected/kube-api-access-2snrg:{mountpoint:/var/lib/kubelet/pods/e0532776-f74d-4d8a-832a-6719d3fec3c2/volumes/kubernetes.io~projected/kube-api-access-2snrg major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0532776-f74d-4d8a-832a-6719d3fec3c2/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/e0532776-f74d-4d8a-832a-6719d3fec3c2/volumes/kubernetes.io~secret/signing-key major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6/volumes/kubernetes.io~projected/kube-api-access-s8tx7:{mountpoint:/var/lib/kubelet/pods/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6/volumes/kubernetes.io~projected/kube-api-access-s8tx7 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f69dcc85-0b97-43eb-8674-0ec1415c8dfb/volumes/kubernetes.io~projected/kube-api-access-rsphz:{mountpoint:/var/lib/kubelet/pods/f69dcc85-0b97-43eb-8674-0ec1415c8dfb/volumes/kubernetes.io~projected/kube-api-access-rsphz major:0 minor:294 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cd346f568b569f8b6ac5a3d34b1dde0c04834a9ceca643b59f96eea3adcccadc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4bb6e4bb383ea644fdb68abe70ced994f8f6edd0b337a0abaee45850c0f64743/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/51682420e640c781f074a9f44a7d4333c6411abbc3470398b4011c09052ddc3a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/40209d72c0df9e3cb3005c9a5fbd05a359ef8af6ff62321697a68deab8537692/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/21b005c719d25a550efead43a9e166476c6ff16fc7108fb2c9495775f9133d5f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e3f7c52e34def885f5201394315685d2469e479b10efb18205d9c0a450130a6e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/202e86f78d7ea020cdb87bc6c75030c4f68f7cab6c48816252755d9f15b77a88/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/121a2b84761f2febb39a0a7f777364c0fab2c7bdf6e52593076470d9da1942ac/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cd265e93be8b99411239900b438bf7772332d5316aaae9599a41c66e1b7e1062/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/70b0c56a40622ae514da0f49b166f1d2ada45a956b7830195062307397fcfe4b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/27d81044c678a58e3171a96d0ff145e2bedb039868f99dffb42ed9d22347e44f/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/90f592aef9d4eb0c48fa6ea060a851bf32a0ea0fd6519d5296112c046020814d/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4986ac2609f9c4b8ec8e9d66156114eb17decb0a611803a7f21da6ba81d27bfa/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/67605ffa6a7d63f72ad435422e889f5b54a8702693f522983a65f350e49a6823/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/58758e1cbf1d03c3b33f27849e3e948922969093c9187407e5e95af3e26357d9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/39992bd134bec2deff0a138eb1e531dcb822ee6a07a14d7d9164c66bb1b9f245/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b311acf30436225d7d52949b279fbef01e5001992345269df3b52de40c1625ce/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/93b334d8f65579285fc48475ceda39bacc349e85c2dfa5a9af89268f40f5d1f4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/02ec77f0b2049eaedbb4b8c725254886049379615a0df1750b57a8c44dbc7d4b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/72a24e7a7e3837ea07afaf3a086faaa602a94d3c419ad69df73225641a30348c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/582d40c42c20db5fb7230afcf9868f36c73ff7a17050145f241d6713eaa5f2af/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/ebac973040ca33431cf77f5442682e7423de9db6b40496bc833aca076a71b589/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/bd3d9b6346586066fe5f4603c1079318d7dc4d62dc5e952169a3adb3856b5391/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f44c50ab95d17b30d9b88b334fcc2c71d46a5d988e658f66c8e878c4f5e95aae/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a646232fa248c3b19279297b5aacc9faacc5b77e13035a447e5af98340af18d6/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/2bba4a70c5145367bb6d3d9a02575f2f274bb137b50cb59ae475169ed2d91c4c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/3a4aea59f95cf7e0ae64048e45e8b17e8f5799346ca3bce4bb3cbcf252dd0cde/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/8732a8e79e09a2e8c99cfc03eaf56caa754f1bd038bb3c6f816aae9ed1eea19c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/27a160480d448af2563ba17ae3b565eb68f95543f5d191fa3e4cccf9037df3f6/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/630afeec2e0575157cb7d9effd47496e29a630199dfaf1893c3df83d2830165d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/835089668b5b0f16805c2b266d8804f2964945c41ac042e169244a68c90dda63/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/2bb87c1d4567cfe0fab3320f21af1174eb4e81a3175cd5d216b3443c22a2237a/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/881fff3d0af7d9a828bb12fb7b81a36a5c5bdb103fc908bb50004045cef6ef71/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/6459de8626741c298e00f3aa0936a25813d65d2c9f62605aacdb20003acdcf19/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/95e836928cdb9b09238517136024e5d588ac588b4b4567437bcb3dcda27099f8/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/acd78cad5aa0b4ef36a129da49eedff57186bd746cc5088946af077d69f537a5/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/1d66c6bb67e40df0011d9b55bbf5f1f65c369c38f87c874b7b855b827d58e7a1/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0b8b7301b3739374e0c6927cc7b9fa5c5fbf535cb20d6a0da978df2e0bfd28e6/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/ded2d4fd4700b41d19ce48de9b5c7ac983f8fa0024c276860b709ece5d2e3ee8/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/bdbf065df96e907a304e446e33e45b9d049e96c5e34b711ac3e6a4984849dab4/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/c5b375d2b909132ba27449036b669b5b3e7e93e34f0423ee1c66e3acd1d25270/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/aecf20de06f179308faf869de2fa21ca8aeb04892306720063d4c34b5f616f41/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/a9bed59f832200797a3cbf490ef645460d1b0f21d808bb975d20c77e01135306/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/94f32808423eb43af4651368ae0af54c4fb44c2fc0d5c9b0495af4db34fd6643/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/a001bb4c3eb1fcd433992d44f0a9b3f2246fb573f756b358c7fe5ad571d9374f/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/f0ec094fed36f5270c5131cc4b29a10c5919bc1bfd31efa486eccfac9089edd7/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/7be56fe812a8bed07a7e9193edcae8a9c77a047fba942f6868d21987406ba9fe/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/cc5896d46bc6bc2cab1fe45a3c82bd7bf2d6d09663aa169aedd5fedf86c8d9f4/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/f750b0aec169b17065ccea2ac0786ccfd116c33e5466ab47e4a7e2247596c830/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/bb16fb0e84f46a5db0ca1b145465e8322849345dcdbbcb1c8f7b3af9da098b79/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/f9e033b6f33be0a9e99a6aea9136a11d62bdb1c09ab1ee39a500d9dcc61315ed/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/0ab470b67a6d7aa16cc0e7dc33b5277420be16c601e2fa858e6c9211f41fce63/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/5122678931fd61e8fc04adef770b158ce5f38c4074ccd1deb2be3ec280b38b5e/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/df40d8edf4b4451206123a9cd6c140d685828f6ec198a47cd882adefe4ad6d49/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/bac011b9983d9ffcfa6d63d01d94c8edef3afb9e2e15ebdee445f1e1277c8e7e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/38e398450fe799d114cd73cbc10c2cdd2b02c8883d2c1c908ed136cf1003c599/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/65a392b793c053576046a8013a63ea4884d9bb54ef2e3ad765a1b34058b3632b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/da7d30b588d8f1b65c3fb66d2fa3982f369d4866ac8cf22191149436f94fc172/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3db7a7f12dc93f3f907c9d2827caa4b8719cccafe9f9fbf97005366773521138/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/71ac048eef25be85e24ead6d9bdcf4419f575a5dc291f39fee5dc2e0918830ae/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c3a00568a3bc1716727753ffe579054df7b45f677a4064eda720c121c1a88036/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8bc0cc0bd9de41c2b960c839c9d1b1df00f369533b075ec245e6feff714e63d9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 00:42:38.515439 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.515042    5104 manager.go:217] Machine: {Timestamp:2026-10-30 00:42:38.5145743 +0000 UTC m=+0.062264806 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3150050 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0d4b17ceaff9a33d0fe954c0bde8 SystemUUID:ec2e0d4b-17ce-aff9-a33d-0fe954c0bde8 BootID:25ef4616-8dbf-466e-9a9b-8e601af465d3 Filesystems:[{Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff41799c38e73f9a43ffc3e6b69a89cbf51f02d1c7b58331dde4a30d502054fe/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33045ca4-6bda-4fdc-9d44-f231e4161fae/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/56ea1b39-0a84-4b31-a200-0c1b68cfc927/volumes/kubernetes.io~projected/kube-api-access-w7qq9 DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96fae5a4031880a78bac9a755c06367fa3a998e276d66611b25f4bb6dcc7e06c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db543397-1d27-4098-a794-7a57aaa7904b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d56b6c9e-b741-42c5-90d7-8c8c641064fa/volumes/kubernetes.io~projected/kube-api-access-xpk5j DeviceMajor:0 DeviceMinor:320 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e0532776-f74d-4d8a-832a-6719d3fec3c2/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4254ef03-935f-47d4-948a-70855cb3afb8/volumes/kubernetes.io~projected/kube-api-access-4l9nl DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405/volumes/kubernetes.io~projected/kube-api-access-4cnf6 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2deae516-c4e0-4377-98ee-084581c4b027/volumes/kubernetes.io~projected/kube-api-access-5pw8p DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b9dac87d-7a94-43b5-ae69-38f6975388ed/volumes/kubernetes.io~projected/kube-api-access-d5tx8 DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c304b3252841b1a3e7bd31eaefd05a3a89bd8c370d1d2af68fd19551e7d71ad4/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/49fcf677-f503-492a-90e8-c95e329ee6f6/volumes/kubernetes.io~projected/kube-api-access-wp67x DeviceMajor:0 DeviceMinor:240 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b819267f7bb1185abf55922a2797864493555e9eff9158feb54b2e7bed5057e/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8fa684a7e915bc9a5737948f2a4f9087640d6ef8ecc4db2b47580327fba7095/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/15ba866d-3fba-4bf9-beec-14ecda492c3f/volumes/kubernetes.io~projected/kube-api-access-pxtt4 DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9063c68f92d2c33f53d770a9dedc6c9e0372507b62d00dda32a636db5999e5aa/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cb4d9d883799c0de98745bc1149013e88d45d035c0c616121bb0b60ea1c0d23/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3bb0053-40b7-49df-9490-192e78a9b720/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a/volumes/kubernetes.io~projected/kube-api-access-bdscw DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~projected/kube-api-access-6sfzl DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e0532776-f74d-4d8a-832a-6719d3fec3c2/volumes/kubernetes.io~projected/kube-api-access-2snrg DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~projected/kube-api-access-qrgrf DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volumes/kubernetes.io~projected/kube-api-access-p62hj DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce14d2cc-a9e8-49af-be37-9b8e6a433c65/volumes/kubernetes.io~projected/kube-api-access-cg87z DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/303442b5c031dfa67e64696ad482f1f3466163bbf51b19d452685f0a7663048e/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d50536e56bdf06f88016b7a66ac227f253b006e76d13557570c101b66efca658/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/586e2881c8d32bdea15368b7446c8b0d8724e12051d75c475edefbf039d431b3/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0240e78b-bcf9-40a0-8707-0f7976d91005/volumes/kubernetes.io~projected/kube-api-access-gql2j DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a4e1b78a880e1c90c59e2af2ac50f73b75de77b0ad0cd58c817ac5dfa706d75/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f69dcc85-0b97-43eb-8674-0ec1415c8dfb/volumes/kubernetes.io~projected/kube-api-access-rsphz DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a7587e96d9aa2ca3dba26ed5500760dd5df7488d4cf688189d237032d408cb2/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/59071108-f1ba-4346-beab-5cb766aecede/volumes/kubernetes.io~projected/kube-api-access-29q9f DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6/volumes/kubernetes.io~projected/kube-api-access-s8tx7 DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9dac87d-7a94-43b5-ae69-38f6975388ed/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/81da025c-1894-43c8-b83a-0b80409656ac/volumes/kubernetes.io~projected/kube-api-access-5dt8t DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/532cd48be8aa6b55ce672db105d7bf6c304417708938946b2921e4ca89175d68/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/927717eb-ca5c-4f65-8a7a-bdc83054080c/volumes/kubernetes.io~projected/kube-api-access-l6b25 DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8d8e7da-34ee-466e-bf3d-48a8a56b722f/volumes/kubernetes.io~projected/kube-api-access-vdgzs DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccc5ef2e03bbc6dbb03dc349ae32c22dcb1a465242ba95c8786edd7f44f44ac3/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c849f30-20d0-4135-a2b2-e5d2daef76fa/volumes/kubernetes.io~projected/kube-api-access-whkbr DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdbb2c8206606e39d673e1c3da53ada1679b600e07a2235916f856cfc32041b6/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7484c3c21e65844f4ea8b48d76affa6dc60ed06e922e34be59b4f17c2b124071/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6a4543fe-837e-4480-b419-974b723e4475/volumes/kubernetes.io~projected/kube-api-access-fd285 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~projected/kube-api-access-rlj97 DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6efccdb7855156c04f2f7b2dff58f10b6436b26bad189db97a65342abc525e16/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/615b46a4c54925eacc7cbf15bec37c7ad0c000436b828c0301bbab676933ecca/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/501238e2345bfe4f9ebfa0008d8f4b12e0993b7905f8f4bad3f7f3393b72ccc7/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79ef76c0-8e88-4b4a-8d6a-83da18b39d78/volumes/kubernetes.io~projected/kube-api-access-55ncq DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebee0ab5c207b2661616886648299b32db2db9f14b4ce0f50312e859067fcc96/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/db543397-1d27-4098-a794-7a57aaa7904b/volumes/kubernetes.io~projected/kube-api-access-ctjfv DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e082e1-ce44-4deb-8036-89d2ae8dcc46/volumes/kubernetes.io~projected/kube-api-access-p7j55 DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3a8ab87de14c8248680e758b9da685aaf1ed8707618e9aab341875632a0f13a/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33045ca4-6bda-4fdc-9d44-f231e4161fae/volumes/kubernetes.io~projected/kube-api-access-sl9qt DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/59071108-f1ba-4346-beab-5cb766aecede/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:238 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~projected/kube-api-access-fzt92 DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/085c11e9-3ae8-4a1d-abe7-edc7dcad9804/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7469affe-9d00-44b3-a833-4ca25b86da5c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cec9dfe-c016-4228-be90-4724120f7260/volumes/kubernetes.io~projected/kube-api-access-z546g DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/44aeb453-223b-4cfa-a838-6798b6b5aa1c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6182d2c6e3c810fa7d0495e0e45b24f1383e7042c185a72d40390c9f5e88668e/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81da025c-1894-43c8-b83a-0b80409656ac/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d1037f0ef4eb537f025b0e04f1a7a749ee07d715e4ff5880c66056112dbc727/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d027727-5c89-44a2-b1b8-20661fd13909/volumes/kubernetes.io~projected/kube-api-access-2r7cp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64e2b2035554c6d65d1cb62e61ebe849dbc427fd3a978173782c3a3df331a3d1/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3a4e1b78a880e1c MacAddress:42:28:15:cf:55:4d Speed:10000 Mtu:8901} {Name:532cd48be8aa6b5 MacAddress:66:eb:50:c4:c5:8d Speed:10000 Mtu:8901} {Name:586e2881c8d32bd MacAddress:5e:da:5f:b1:5c:bf Speed:10000 Mtu:8901} {Name:6182d2c6e3c810f MacAddress:a2:ce:88:6a:b2:22 Speed:10000 Mtu:8901} {Name:6a7587e96d9aa2c MacAddress:e6:dc:ae:e2:21:28 Speed:10000 Mtu:8901} {Name:6cb4d9d883799c0 MacAddress:72:a5:13:a9:8b:71 Speed:10000 Mtu:8901} {Name:6efccdb7855156c MacAddress:f6:ac:80:dd:b6:53 Speed:10000 Mtu:8901} {Name:7484c3c21e65844 MacAddress:06:75:13:bb:e4:ac Speed:10000 Mtu:8901} {Name:8b819267f7bb118 MacAddress:da:64:91:eb:01:c5 Speed:10000 Mtu:8901} {Name:8d1037f0ef4eb53 MacAddress:7a:51:ae:f1:5f:64 Speed:10000 Mtu:8901} {Name:9063c68f92d2c33 MacAddress:9e:26:55:84:d1:48 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e2:79:1c:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:f9:9b:4d:e5:c8 Speed:0 Mtu:8901} {Name:c304b3252841b1a MacAddress:82:dc:9f:b1:0f:86 Speed:10000 Mtu:8901} {Name:c3a8ab87de14c82 MacAddress:12:f9:b4:49:94:fe Speed:10000 Mtu:8901} {Name:ccc5ef2e03bbc6d MacAddress:c6:a0:00:97:db:39 Speed:10000 Mtu:8901} {Name:ebee0ab5c207b26 MacAddress:26:32:2f:3f:92:db Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e2:79:1c:55 Speed:0 Mtu:9001} {Name:ff41799c38e73f9 MacAddress:16:b5:9c:07:04:8a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:14:f2:cf:8e:0a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:d9:ad:ec:a7:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:42:38.515439 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.515440    5104 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:42:38.515793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.515520    5104 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:42:38.515867 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.515832    5104 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:42:38.516007 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.515868    5104 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-169.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:42:38.516051 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516009    5104 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:42:38.516051 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516018    5104 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:42:38.516051 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516028    5104 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:42:38.516051 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516042    5104 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:42:38.516160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516063    5104 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:42:38.516262 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516252    5104 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:42:38.516371 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516362    5104 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:42:38.516408 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516398    5104 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:42:38.516438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516408    5104 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:42:38.516438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516417    5104 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:42:38.516438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516428    5104 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:42:38.516438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.516437    5104 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:42:38.517286 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.517264    5104 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:42:38.517286 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.517278    5104 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:42:38.517886 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.517863    5104 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:42:38.520163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520149    5104 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:42:38.520163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520163    5104 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:42:38.520284 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520274    5104 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:42:38.520512 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520500    5104 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:42:38.520545 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520530    5104 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:42:38.520679 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520669    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:42:38.520714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520685    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:42:38.520714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520691    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:42:38.520714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520697    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:42:38.520793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520737    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:42:38.520793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520745    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:42:38.520793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520751    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:42:38.520793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520757    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:42:38.520793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520763    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:42:38.520793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520771    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:42:38.520793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520787    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:42:38.520979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520798    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:42:38.520979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520827    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:42:38.520979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.520836    5104 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:42:38.524644 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.524627    5104 server.go:1263] "Started kubelet"
Oct 30 00:42:38.524807 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.524749    5104 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:42:38.525080 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.524698    5104 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:42:38.525175 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.525084    5104 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:42:38.525207 ip-10-0-130-169 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:42:38.525422 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.525398    5104 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.529689    5104 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.530168    5104 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.530180    5104 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.530239    5104 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.530262    5104 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.530364    5104 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:36:57 +0000 UTC" deadline="2028-05-11 09:41:06.666879489 +0000 UTC"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.530399    5104 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13424h58m28.136484187s"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:38.530396    5104 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-169.ec2.internal\" not found"
Oct 30 00:42:38.533228 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.530758    5104 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:42:38.533853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.533523    5104 factory.go:153] Registering CRI-O factory
Oct 30 00:42:38.533853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.533539    5104 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:42:38.533853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.533598    5104 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:42:38.533853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.533614    5104 factory.go:55] Registering systemd factory
Oct 30 00:42:38.533853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.533622    5104 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:42:38.533853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.533639    5104 factory.go:103] Registering Raw factory
Oct 30 00:42:38.533853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.533652    5104 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:42:38.534054 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.534006    5104 manager.go:319] Starting recovery of all containers
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.552984    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81da025c-1894-43c8-b83a-0b80409656ac" volumeName="kubernetes.io/secret/81da025c-1894-43c8-b83a-0b80409656ac-serving-cert" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553028    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db543397-1d27-4098-a794-7a57aaa7904b" volumeName="kubernetes.io/secret/db543397-1d27-4098-a794-7a57aaa7904b-serving-cert" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553039    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6" volumeName="kubernetes.io/projected/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-kube-api-access-s8tx7" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553048    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44aeb453-223b-4cfa-a838-6798b6b5aa1c" volumeName="kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-stats-auth" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553060    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ef76c0-8e88-4b4a-8d6a-83da18b39d78" volumeName="kubernetes.io/projected/79ef76c0-8e88-4b4a-8d6a-83da18b39d78-kube-api-access-55ncq" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553069    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="927717eb-ca5c-4f65-8a7a-bdc83054080c" volumeName="kubernetes.io/projected/927717eb-ca5c-4f65-8a7a-bdc83054080c-kube-api-access-l6b25" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553076    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6" volumeName="kubernetes.io/configmap/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-serviceca" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553085    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="085c11e9-3ae8-4a1d-abe7-edc7dcad9804" volumeName="kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-env-overrides" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553097    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="085c11e9-3ae8-4a1d-abe7-edc7dcad9804" volumeName="kubernetes.io/secret/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553105    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2296d29f-a76e-4a37-bf06-34a77068f571" volumeName="kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553116    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49fcf677-f503-492a-90e8-c95e329ee6f6" volumeName="kubernetes.io/projected/49fcf677-f503-492a-90e8-c95e329ee6f6-kube-api-access-wp67x" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553123    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56ea1b39-0a84-4b31-a200-0c1b68cfc927" volumeName="kubernetes.io/projected/56ea1b39-0a84-4b31-a200-0c1b68cfc927-kube-api-access-w7qq9" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553131    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7469affe-9d00-44b3-a833-4ca25b86da5c" volumeName="kubernetes.io/configmap/7469affe-9d00-44b3-a833-4ca25b86da5c-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553139    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7469affe-9d00-44b3-a833-4ca25b86da5c" volumeName="kubernetes.io/secret/7469affe-9d00-44b3-a833-4ca25b86da5c-agent-certs" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553149    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d027727-5c89-44a2-b1b8-20661fd13909" volumeName="kubernetes.io/empty-dir/7d027727-5c89-44a2-b1b8-20661fd13909-tmp-dir" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553157    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="085c11e9-3ae8-4a1d-abe7-edc7dcad9804" volumeName="kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553170    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="085c11e9-3ae8-4a1d-abe7-edc7dcad9804" volumeName="kubernetes.io/projected/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-kube-api-access-p62hj" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553179    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2296d29f-a76e-4a37-bf06-34a77068f571" volumeName="kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553187    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33045ca4-6bda-4fdc-9d44-f231e4161fae" volumeName="kubernetes.io/projected/33045ca4-6bda-4fdc-9d44-f231e4161fae-kube-api-access-sl9qt" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553195    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" volumeName="kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553202    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" volumeName="kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553210    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" volumeName="kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553218    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" volumeName="kubernetes.io/projected/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-kube-api-access-vdgzs" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553227    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ba866d-3fba-4bf9-beec-14ecda492c3f" volumeName="kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553235    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ee8ed6-aae3-42a9-adfe-3ecbfd348141" volumeName="kubernetes.io/projected/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-kube-api-access-qrgrf" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553249    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33045ca4-6bda-4fdc-9d44-f231e4161fae" volumeName="kubernetes.io/configmap/33045ca4-6bda-4fdc-9d44-f231e4161fae-config" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553258    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cec9dfe-c016-4228-be90-4724120f7260" volumeName="kubernetes.io/projected/9cec9dfe-c016-4228-be90-4724120f7260-kube-api-access-z546g" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553266    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ee8ed6-aae3-42a9-adfe-3ecbfd348141" volumeName="kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-tuned" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553278    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0532776-f74d-4d8a-832a-6719d3fec3c2" volumeName="kubernetes.io/secret/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-key" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553298    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69dcc85-0b97-43eb-8674-0ec1415c8dfb" volumeName="kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553307    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="927717eb-ca5c-4f65-8a7a-bdc83054080c" volumeName="kubernetes.io/configmap/927717eb-ca5c-4f65-8a7a-bdc83054080c-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553314    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db543397-1d27-4098-a794-7a57aaa7904b" volumeName="kubernetes.io/projected/db543397-1d27-4098-a794-7a57aaa7904b-kube-api-access-ctjfv" seLinuxMountContext=""
Oct 30 00:42:38.553285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553325    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ba866d-3fba-4bf9-beec-14ecda492c3f" volumeName="kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553339    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44aeb453-223b-4cfa-a838-6798b6b5aa1c" volumeName="kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-default-certificate" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553365    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" volumeName="kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553378    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81da025c-1894-43c8-b83a-0b80409656ac" volumeName="kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-service-ca-bundle" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553391    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81da025c-1894-43c8-b83a-0b80409656ac" volumeName="kubernetes.io/projected/81da025c-1894-43c8-b83a-0b80409656ac-kube-api-access-5dt8t" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553406    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d56b6c9e-b741-42c5-90d7-8c8c641064fa" volumeName="kubernetes.io/projected/d56b6c9e-b741-42c5-90d7-8c8c641064fa-kube-api-access-xpk5j" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553420    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0240e78b-bcf9-40a0-8707-0f7976d91005" volumeName="kubernetes.io/configmap/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-config" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553432    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c849f30-20d0-4135-a2b2-e5d2daef76fa" volumeName="kubernetes.io/projected/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-api-access-whkbr" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553450    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405" volumeName="kubernetes.io/projected/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405-kube-api-access-4cnf6" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553476    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49fcf677-f503-492a-90e8-c95e329ee6f6" volumeName="kubernetes.io/empty-dir/49fcf677-f503-492a-90e8-c95e329ee6f6-tmp-dir" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553490    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" volumeName="kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553503    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ba866d-3fba-4bf9-beec-14ecda492c3f" volumeName="kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553517    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2296d29f-a76e-4a37-bf06-34a77068f571" volumeName="kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553530    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" volumeName="kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553544    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" volumeName="kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553556    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e082e1-ce44-4deb-8036-89d2ae8dcc46" volumeName="kubernetes.io/projected/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kube-api-access-p7j55" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553569    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9dac87d-7a94-43b5-ae69-38f6975388ed" volumeName="kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-trusted-ca" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553599    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db543397-1d27-4098-a794-7a57aaa7904b" volumeName="kubernetes.io/configmap/db543397-1d27-4098-a794-7a57aaa7904b-config" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553611    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0532776-f74d-4d8a-832a-6719d3fec3c2" volumeName="kubernetes.io/configmap/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-cabundle" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553623    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81da025c-1894-43c8-b83a-0b80409656ac" volumeName="kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-snapshots" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553639    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2296d29f-a76e-4a37-bf06-34a77068f571" volumeName="kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553653    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2deae516-c4e0-4377-98ee-084581c4b027" volumeName="kubernetes.io/configmap/2deae516-c4e0-4377-98ee-084581c4b027-telemetry-config" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553667    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254ef03-935f-47d4-948a-70855cb3afb8" volumeName="kubernetes.io/projected/4254ef03-935f-47d4-948a-70855cb3afb8-kube-api-access-4l9nl" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553683    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44aeb453-223b-4cfa-a838-6798b6b5aa1c" volumeName="kubernetes.io/projected/44aeb453-223b-4cfa-a838-6798b6b5aa1c-kube-api-access-fzt92" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553719    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49fcf677-f503-492a-90e8-c95e329ee6f6" volumeName="kubernetes.io/configmap/49fcf677-f503-492a-90e8-c95e329ee6f6-config-volume" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553733    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4543fe-837e-4480-b419-974b723e4475" volumeName="kubernetes.io/projected/6a4543fe-837e-4480-b419-974b723e4475-kube-api-access-fd285" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553757    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="085c11e9-3ae8-4a1d-abe7-edc7dcad9804" volumeName="kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-config" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553779    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ba866d-3fba-4bf9-beec-14ecda492c3f" volumeName="kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553799    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d027727-5c89-44a2-b1b8-20661fd13909" volumeName="kubernetes.io/projected/7d027727-5c89-44a2-b1b8-20661fd13909-kube-api-access-2r7cp" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553821    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9dac87d-7a94-43b5-ae69-38f6975388ed" volumeName="kubernetes.io/projected/b9dac87d-7a94-43b5-ae69-38f6975388ed-kube-api-access-d5tx8" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553842    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" volumeName="kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553863    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81da025c-1894-43c8-b83a-0b80409656ac" volumeName="kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-tmp" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553886    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" volumeName="kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553898    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9dac87d-7a94-43b5-ae69-38f6975388ed" volumeName="kubernetes.io/secret/b9dac87d-7a94-43b5-ae69-38f6975388ed-serving-cert" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553910    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0532776-f74d-4d8a-832a-6719d3fec3c2" volumeName="kubernetes.io/projected/e0532776-f74d-4d8a-832a-6719d3fec3c2-kube-api-access-2snrg" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553924    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33045ca4-6bda-4fdc-9d44-f231e4161fae" volumeName="kubernetes.io/secret/33045ca4-6bda-4fdc-9d44-f231e4161fae-serving-cert" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553935    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59071108-f1ba-4346-beab-5cb766aecede" volumeName="kubernetes.io/secret/59071108-f1ba-4346-beab-5cb766aecede-cert" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553949    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ee8ed6-aae3-42a9-adfe-3ecbfd348141" volumeName="kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-tmp" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553963    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0240e78b-bcf9-40a0-8707-0f7976d91005" volumeName="kubernetes.io/projected/0240e78b-bcf9-40a0-8707-0f7976d91005-kube-api-access-gql2j" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553976    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c849f30-20d0-4135-a2b2-e5d2daef76fa" volumeName="kubernetes.io/empty-dir/1c849f30-20d0-4135-a2b2-e5d2daef76fa-data-volume" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.553989    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2296d29f-a76e-4a37-bf06-34a77068f571" volumeName="kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554003    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2296d29f-a76e-4a37-bf06-34a77068f571" volumeName="kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554015    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2deae516-c4e0-4377-98ee-084581c4b027" volumeName="kubernetes.io/projected/2deae516-c4e0-4377-98ee-084581c4b027-kube-api-access-5pw8p" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554028    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59071108-f1ba-4346-beab-5cb766aecede" volumeName="kubernetes.io/projected/59071108-f1ba-4346-beab-5cb766aecede-kube-api-access-29q9f" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554054    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81da025c-1894-43c8-b83a-0b80409656ac" volumeName="kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554067    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85585df9-d495-4d44-bd82-398d44b76cf8" volumeName="kubernetes.io/configmap/85585df9-d495-4d44-bd82-398d44b76cf8-nginx-conf" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554079    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2296d29f-a76e-4a37-bf06-34a77068f571" volumeName="kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554093    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254ef03-935f-47d4-948a-70855cb3afb8" volumeName="kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554110    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254ef03-935f-47d4-948a-70855cb3afb8" volumeName="kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554123    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" volumeName="kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554136    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3bb0053-40b7-49df-9490-192e78a9b720" volumeName="kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554151    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9dac87d-7a94-43b5-ae69-38f6975388ed" volumeName="kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-config" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554162    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4302e4-1ec4-4ce1-a136-a91a5d1d306a" volumeName="kubernetes.io/projected/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a-kube-api-access-bdscw" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554177    5104 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce14d2cc-a9e8-49af-be37-9b8e6a433c65" volumeName="kubernetes.io/projected/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-kube-api-access-cg87z" seLinuxMountContext=""
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554189    5104 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:42:38.554802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.554196    5104 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:42:38.562284 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.562258    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/1.log"
Oct 30 00:42:38.563438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.563409    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/1.log"
Oct 30 00:42:38.566935 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.566903    5104 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:42:38.568492 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.568473    5104 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:42:38.568601 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.568591    5104 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:42:38.568713 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.568694    5104 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:42:38.568837 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:38.568743    5104 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:42:38.577760 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.577736    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:38.578115 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.578095    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:38.582805 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.582773    5104 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:42:38.582805 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.582775    5104 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:42:38.594493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.594446    5104 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:42:38.594493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.594477    5104 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:42:38.598945 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.598925    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/1.log"
Oct 30 00:42:38.599040 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.598952    5104 generic.go:356] "Generic (PLEG): container finished" podID="b9dac87d-7a94-43b5-ae69-38f6975388ed" containerID="f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14" exitCode=255
Oct 30 00:42:38.606606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.606577    5104 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:42:38.612394 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.612362    5104 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="b8d9aba6ae230089b9fee539d199dd556930b6a1000e985a1e76002436377b0a" exitCode=0
Oct 30 00:42:38.612394 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.612393    5104 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="11e9d572e955dd0c777d48fed519d371dbf8aa0dcb35c676d4b120704c2c88fa" exitCode=0
Oct 30 00:42:38.612603 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.612403    5104 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="a1f2be932b4253e40b54d7df954050a09f0cee352fc141d5c9f839afb482c3a7" exitCode=0
Oct 30 00:42:38.612603 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.612412    5104 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="ccdbfc1aead2a2aaf5ae26f1f63e6b620332f252115b738a44b3b3ce3197ba38" exitCode=0
Oct 30 00:42:38.612603 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.612420    5104 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="4c6582d5c5677ebd9ce4950331a8c013299566ee288b5df42f1d0e742f773e32" exitCode=0
Oct 30 00:42:38.612603 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.612428    5104 generic.go:356] "Generic (PLEG): container finished" podID="a8d8e7da-34ee-466e-bf3d-48a8a56b722f" containerID="007d3b7c9e5235979299802e40b921d51d90388ec948ae537fa85d9a7fb7b643" exitCode=0
Oct 30 00:42:38.616436 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.616416    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:42:38.616846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.616819    5104 generic.go:356] "Generic (PLEG): container finished" podID="085c11e9-3ae8-4a1d-abe7-edc7dcad9804" containerID="09d4982670b7f504b69df58781cf46234e80f64aef14260654a5a2404674367f" exitCode=1
Oct 30 00:42:38.623861 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.623836    5104 generic.go:356] "Generic (PLEG): container finished" podID="3bf851886b8d1abdda99ba2c15c29f78" containerID="6ddd8e6ee8ea0407bb055951087213f6692859f25a5b8f1e0a844f7e06660510" exitCode=0
Oct 30 00:42:38.652221 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.652158    5104 manager.go:324] Recovery completed
Oct 30 00:42:38.668862 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:38.668821    5104 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 00:42:38.674989 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.674966    5104 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:42:38.674989 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.674980    5104 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675001    5104 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675130    5104 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675140    5104 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675155    5104 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675160    5104 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675166    5104 policy_none.go:50] "Start"
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675174    5104 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:42:38.675260 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675183    5104 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:42:38.675802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675786    5104 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:42:38.675802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675802    5104 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:42:38.675865 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.675808    5104 policy_none.go:44] "Start"
Oct 30 00:42:38.680349 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.680321    5104 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:42:38.685058 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685045    5104 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:42:38.685112 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:38.685089    5104 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:42:38.685112 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685102    5104 server.go:87] "Starting device plugin registration server"
Oct 30 00:42:38.685432 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685420    5104 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:42:38.685495 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685434    5104 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:42:38.685615 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685595    5104 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:42:38.685843 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685788    5104 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:42:38.685843 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685839    5104 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:42:38.685843 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685846    5104 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:42:38.686037 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.685900    5104 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:42:38.685821213Z","UUID":"b514af25-acbe-4bb5-86fb-7ff77dc8b94e","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:42:38.687197 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.687179    5104 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:42:38.687288 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.687203    5104 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:42:38.687730 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:38.687708    5104 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:42:38.786846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.786810    5104 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:42:38.788678 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.788646    5104 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:42:38.788778 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.788685    5104 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:42:38.788778 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.788697    5104 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-169.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:42:38.788778 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.788736    5104 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.821029 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.821002    5104 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.821134 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.821082    5104 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.869745 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.869677    5104 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"]
Oct 30 00:42:38.870370 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.870332    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal" event={"ID":"c2831292250175cd4fa5c7c5f1d89a07","Type":"ContainerStarted","Data":"b30bc04a6379ceb2e5edcbf2d797e6583b182e18d4e6409b2e287d7594e9732c"}
Oct 30 00:42:38.870424 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.870378    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal" event={"ID":"c2831292250175cd4fa5c7c5f1d89a07","Type":"ContainerStarted","Data":"37140312bd3b03f6ff10dc60490c33aec503596c6f8a6c63d35cb27d6c17b47b"}
Oct 30 00:42:38.870495 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.870478    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal" event={"ID":"3bf851886b8d1abdda99ba2c15c29f78","Type":"ContainerStarted","Data":"da89aca54a7cdfdc39808f2060333ad707928c837c2a02918bca422ab42d05d9"}
Oct 30 00:42:38.870564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.870498    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal" event={"ID":"3bf851886b8d1abdda99ba2c15c29f78","Type":"ContainerDied","Data":"6ddd8e6ee8ea0407bb055951087213f6692859f25a5b8f1e0a844f7e06660510"}
Oct 30 00:42:38.870564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.870513    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal" event={"ID":"3bf851886b8d1abdda99ba2c15c29f78","Type":"ContainerStarted","Data":"19401f0eae4f91e2da173adcecd427920cac52e416fa606ffb8c716d2434661d"}
Oct 30 00:42:38.870564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.870520    5104 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.870693 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.870597    5104 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.892177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.892153    5104 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:42:38.892266 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:38.892196    5104 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.892572 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.892558    5104 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:42:38.892608 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:38.892600    5104 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.996113 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.996077    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2831292250175cd4fa5c7c5f1d89a07-config\") pod \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" (UID: \"c2831292250175cd4fa5c7c5f1d89a07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.996273 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.996117    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:38.996273 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:38.996143    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.096547 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.096510    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2831292250175cd4fa5c7c5f1d89a07-config\") pod \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" (UID: \"c2831292250175cd4fa5c7c5f1d89a07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.096747 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.096558    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.096747 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.096616    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.096747 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.096629    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2831292250175cd4fa5c7c5f1d89a07-config\") pod \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" (UID: \"c2831292250175cd4fa5c7c5f1d89a07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.096747 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.096696    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.096938 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.096809    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bf851886b8d1abdda99ba2c15c29f78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" (UID: \"3bf851886b8d1abdda99ba2c15c29f78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.517661 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.517623    5104 apiserver.go:52] "Watching apiserver"
Oct 30 00:42:39.566952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.566913    5104 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:42:39.567794 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.567753    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-77hhk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z","openshift-cluster-node-tuning-operator/tuned-ztnp7","openshift-image-registry/image-registry-54566cf8bb-wxl8f","openshift-insights/insights-operator-7c4986d449-v9wjf","kube-system/kas-connection-checker-685895c7c9-rrv72","openshift-insights/insights-runtime-extractor-lmwj7","openshift-ingress-canary/ingress-canary-26wcv","openshift-ovn-kubernetes/ovnkube-node-tntr4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal","openshift-image-registry/image-registry-7488dfd947-qp2j9","openshift-network-operator/iptables-alerter-6ns76","openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f","kube-system/konnectivity-agent-skbfn","openshift-insights/periodic-gathering-wmj68-h5vbk","kube-system/kas-connection-checker-685895c7c9-xf7xr","openshift-multus/network-metrics-daemon-2q7g6","openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l","openshift-console-operator/console-operator-7cd56b8dbc-5phkz","openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm","openshift-multus/multus-additional-cni-plugins-b6zxp","openshift-ingress/router-default-75647bc475-s8spf","openshift-network-console/networking-console-plugin-86c478bb88-s248j","kube-system/kas-connection-checker-685895c7c9-ss447","openshift-dns/dns-default-dkpn9","openshift-service-ca/service-ca-5fffc97bdf-q48h7","openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7","openshift-image-registry/node-ca-r6gln","openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq","kube-system/global-pull-secret-syncer-nk4k6","openshift-multus/multus-tcqrb","kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal","openshift-dns/node-resolver-rxj86","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"]
Oct 30 00:42:39.569638 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.569613    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:39.573703 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.573667    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.573703 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.573694    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.574349 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.574315    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.575372 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.575352    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:39.576827 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.576808    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:39.581395 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.581374    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.581768 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.581751    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.582003 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.581986    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.582003 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.581999    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.582552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.582533    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.582650 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.582576    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.583314 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.583294    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.583876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.583849    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.583978 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.583961    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 00:42:39.584134 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584114    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.584237 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584183    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.584237 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584210    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.584237 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584215    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.584237 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584222    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:42:39.584413 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584246    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.584413 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584269    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 00:42:39.584413 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584284    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 00:42:39.584413 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584409    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m75ds\""
Oct 30 00:42:39.584696 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584579    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7ssb7\""
Oct 30 00:42:39.584696 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584581    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-v9ptc\""
Oct 30 00:42:39.584696 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584597    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 00:42:39.584696 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584692    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.584867 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584740    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cw5zq\""
Oct 30 00:42:39.584867 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.584797    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-chgph\""
Oct 30 00:42:39.594160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.594134    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 00:42:39.598235 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.598209    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:42:39.598403 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.598382    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:42:39.598555 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.598540    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.598627 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.598615    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 00:42:39.598744 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.598731    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 00:42:39.598864 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.598851    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wp9zh\""
Oct 30 00:42:39.598974 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.598961    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5nrg4\""
Oct 30 00:42:39.599090 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599077    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-stkll\""
Oct 30 00:42:39.599166 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599154    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.599272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599260    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\""
Oct 30 00:42:39.599420 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599405    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hrfmc\""
Oct 30 00:42:39.599771 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599744    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 00:42:39.599914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599766    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.600030 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599918    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.600030 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599874    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 00:42:39.600030 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599958    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.600030 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599785    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:42:39.600030 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.599950    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:42:39.600269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600067    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.600269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600128    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 00:42:39.600269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600190    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 00:42:39.600269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600261    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:42:39.600452 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600398    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\""
Oct 30 00:42:39.600452 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600437    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:42:39.600584 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600497    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:42:39.600584 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600567    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.600584 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600560    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.600708 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600597    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:42:39.600708 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600656    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.601011 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.600994    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfrd4\""
Oct 30 00:42:39.601064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.601037    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 00:42:39.607174 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607139    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:42:39.607311 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607225    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 00:42:39.607311 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607286    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.607311 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607290    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:42:39.607311 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607304    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:42:39.607556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607482    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.607556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607518    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fmgv7\""
Oct 30 00:42:39.607556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607531    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 00:42:39.607556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607538    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607570    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607572    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607581    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607589    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607608    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607658    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607673    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607696    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607694    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607749    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607769    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607792    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.607816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607814    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607827    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607884    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607915    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607931    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.607946    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608026    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608026    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608035    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608036    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608072    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608151    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ph2zs\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608184    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608185    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cd6zv\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608208    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hp8gq\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608222    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-smdbv\""
Oct 30 00:42:39.608341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608331    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ppwp\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608353    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608368    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608392    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tgxx\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608394    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tlfwv\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608483    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608515    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2cb6h\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608548    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gz8r\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608566    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnsxr\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608687    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608699    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608728    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608729    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608747    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b5pp6\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608766    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608821    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608832    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7mtf\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608866    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608876    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fmf46\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608877    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608904    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:42:39.608941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.608928    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 00:42:39.609857 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.609845    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:42:39.609908 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.609896    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kbdjq\""
Oct 30 00:42:39.632880 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.632855    5104 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:42:39.633554 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.633539    5104 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.633679 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.633665    5104 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.653069 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.653043    5104 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:42:39.653193 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.653084    5104 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-169.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.653496 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.653469    5104 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:42:39.653592 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.653503    5104 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-169.ec2.internal"
Oct 30 00:42:39.699975 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.699941    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-run\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.699975 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.699976    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-tuned\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.700191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.699994    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-system-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.700191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700010    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-netns\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.700191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700057    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-netd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.700191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700100    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-env-overrides\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.700191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700117    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-tuned\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.700398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700188    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.700398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700277    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9nl\" (UniqueName: \"kubernetes.io/projected/4254ef03-935f-47d4-948a-70855cb3afb8-kube-api-access-4l9nl\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.700398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700391    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjfv\" (UniqueName: \"kubernetes.io/projected/db543397-1d27-4098-a794-7a57aaa7904b-kube-api-access-ctjfv\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:39.700572 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700506    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d027727-5c89-44a2-b1b8-20661fd13909-hosts-file\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:42:39.700572 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700533    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.700572 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700542    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-env-overrides\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.700698 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700617    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-slash\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.700698 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700636    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-var-lib-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.700698 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700682    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.700789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700710    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.700789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700763    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:39.700910 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700891    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.700951 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700932    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-device-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.700951 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700939    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.701028 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700952    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59071108-f1ba-4346-beab-5cb766aecede-cert\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:39.701028 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.700977    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-default-certificate\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.701028 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701001    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp67x\" (UniqueName: \"kubernetes.io/projected/49fcf677-f503-492a-90e8-c95e329ee6f6-kube-api-access-wp67x\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.701141 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701029    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.701141 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701072    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cnibin\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.701141 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701098    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-config\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.701141 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701121    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701147    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpk5j\" (UniqueName: \"kubernetes.io/projected/d56b6c9e-b741-42c5-90d7-8c8c641064fa-kube-api-access-xpk5j\") pod \"migrator-66df5d496b-2kqhm\" (UID: \"d56b6c9e-b741-42c5-90d7-8c8c641064fa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701155    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59071108-f1ba-4346-beab-5cb766aecede-cert\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701173    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701204    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701206    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701229    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-system-cni-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701255    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-config\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701278    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701300    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-systemd-units\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.701316 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701302    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-default-certificate\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701326    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ncq\" (UniqueName: \"kubernetes.io/projected/79ef76c0-8e88-4b4a-8d6a-83da18b39d78-kube-api-access-55ncq\") pod \"kas-connection-checker-685895c7c9-xf7xr\" (UID: \"79ef76c0-8e88-4b4a-8d6a-83da18b39d78\") " pod="kube-system/kas-connection-checker-685895c7c9-xf7xr"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701328    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-config\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701380    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgzs\" (UniqueName: \"kubernetes.io/projected/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-kube-api-access-vdgzs\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701516    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-node-log\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701574    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pw8p\" (UniqueName: \"kubernetes.io/projected/2deae516-c4e0-4377-98ee-084581c4b027-kube-api-access-5pw8p\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701606    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-config\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701651    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701679    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-host\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701706    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnf6\" (UniqueName: \"kubernetes.io/projected/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405-kube-api-access-4cnf6\") pod \"kas-connection-checker-685895c7c9-ss447\" (UID: \"2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405\") " pod="kube-system/kas-connection-checker-685895c7c9-ss447"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701724    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.701801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701760    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701911    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-cnibin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701933    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7469affe-9d00-44b3-a833-4ca25b86da5c-konnectivity-ca\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701950    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85585df9-d495-4d44-bd82-398d44b76cf8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.701977    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-conf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702001    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-host\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702023    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-kubelet\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702046    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-bin\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702097    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-script-lib\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702139    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85585df9-d495-4d44-bd82-398d44b76cf8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:39.702156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702138    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702171    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702188    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7469affe-9d00-44b3-a833-4ca25b86da5c-konnectivity-ca\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702222    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-modprobe-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702251    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c849f30-20d0-4135-a2b2-e5d2daef76fa-data-volume\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702276    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzt92\" (UniqueName: \"kubernetes.io/projected/44aeb453-223b-4cfa-a838-6798b6b5aa1c-kube-api-access-fzt92\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702291    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49fcf677-f503-492a-90e8-c95e329ee6f6-config-volume\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702363    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702372    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovnkube-script-lib\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702395    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c849f30-20d0-4135-a2b2-e5d2daef76fa-data-volume\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702376    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9qt\" (UniqueName: \"kubernetes.io/projected/33045ca4-6bda-4fdc-9d44-f231e4161fae-kube-api-access-sl9qt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702446    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qq9\" (UniqueName: \"kubernetes.io/projected/56ea1b39-0a84-4b31-a200-0c1b68cfc927-kube-api-access-w7qq9\") pod \"network-check-source-6f78d59f99-fkqq5\" (UID: \"56ea1b39-0a84-4b31-a200-0c1b68cfc927\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702495    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702501    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49fcf677-f503-492a-90e8-c95e329ee6f6-config-volume\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702521    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2deae516-c4e0-4377-98ee-084581c4b027-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702552    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-sys\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.702577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702577    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702630    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-sys-fs\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702668    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd285\" (UniqueName: \"kubernetes.io/projected/6a4543fe-837e-4480-b419-974b723e4475-kube-api-access-fd285\") pod \"kas-connection-checker-685895c7c9-rrv72\" (UID: \"6a4543fe-837e-4480-b419-974b723e4475\") " pod="kube-system/kas-connection-checker-685895c7c9-rrv72"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702694    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-snapshots\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702719    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2deae516-c4e0-4377-98ee-084581c4b027-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702767    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702766    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-snapshots\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702801    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/927717eb-ca5c-4f65-8a7a-bdc83054080c-iptables-alerter-script\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702830    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33045ca4-6bda-4fdc-9d44-f231e4161fae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702856    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702890    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702920    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-cni-binary-copy\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702953    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-netns\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.702980    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7j55\" (UniqueName: \"kubernetes.io/projected/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kube-api-access-p7j55\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703014    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703033    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3bb0053-40b7-49df-9490-192e78a9b720-original-pull-secret\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703033    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/927717eb-ca5c-4f65-8a7a-bdc83054080c-iptables-alerter-script\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703038    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-dbus\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703085    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.703150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703105    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgrf\" (UniqueName: \"kubernetes.io/projected/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-kube-api-access-qrgrf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703168    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tx8\" (UniqueName: \"kubernetes.io/projected/b9dac87d-7a94-43b5-ae69-38f6975388ed-kube-api-access-d5tx8\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703248    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-cni-binary-copy\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703321    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-var-lib-kubelet\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-socket-dir-parent\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703370    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703386    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-k8s-cni-cncf-io\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703510    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db543397-1d27-4098-a794-7a57aaa7904b-serving-cert\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703537    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-socket-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703567    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81da025c-1894-43c8-b83a-0b80409656ac-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703580    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8tx7\" (UniqueName: \"kubernetes.io/projected/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-kube-api-access-s8tx7\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703630    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovn-node-metrics-cert\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703710    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62hj\" (UniqueName: \"kubernetes.io/projected/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-kube-api-access-p62hj\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703717    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db543397-1d27-4098-a794-7a57aaa7904b-serving-cert\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703768    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703751    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703833    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703841    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703865    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6b25\" (UniqueName: \"kubernetes.io/projected/927717eb-ca5c-4f65-8a7a-bdc83054080c-kube-api-access-l6b25\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703897    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703929    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-ovn-node-metrics-cert\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703957    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db543397-1d27-4098-a794-7a57aaa7904b-config\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.703995    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2snrg\" (UniqueName: \"kubernetes.io/projected/e0532776-f74d-4d8a-832a-6719d3fec3c2-kube-api-access-2snrg\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704014    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-trusted-ca\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704041    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704071    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704107    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-stats-auth\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.704136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704149    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fcf677-f503-492a-90e8-c95e329ee6f6-tmp-dir\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704213    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81da025c-1894-43c8-b83a-0b80409656ac-serving-cert\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704243    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db543397-1d27-4098-a794-7a57aaa7904b-config\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704244    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-systemd\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704328    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fcf677-f503-492a-90e8-c95e329ee6f6-tmp-dir\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-lib-modules\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704403    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-stats-auth\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704386    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704476    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-multus\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704511    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81da025c-1894-43c8-b83a-0b80409656ac-serving-cert\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704514    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-log-socket\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704585    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704618    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c849f30-20d0-4135-a2b2-e5d2daef76fa-crio-socket\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704643    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-kubelet\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704670    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-multus-certs\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704702    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt8t\" (UniqueName: \"kubernetes.io/projected/81da025c-1894-43c8-b83a-0b80409656ac-kube-api-access-5dt8t\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704733    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlj97\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704759    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-bin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704791    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704815    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9dac87d-7a94-43b5-ae69-38f6975388ed-trusted-ca\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.705064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.704818    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d027727-5c89-44a2-b1b8-20661fd13909-tmp-dir\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705135    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d027727-5c89-44a2-b1b8-20661fd13909-tmp-dir\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705170    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfzl\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705208    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-kubernetes\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705293    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-conf-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705328    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-multus-daemon-config\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-etc-kubernetes\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705411    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-serviceca\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705438    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705579    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/9cec9dfe-c016-4228-be90-4724120f7260-kube-api-access-z546g\") pod \"volume-data-source-validator-58c9f5f46d-56c4z\" (UID: \"9cec9dfe-c016-4228-be90-4724120f7260\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705598    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4254ef03-935f-47d4-948a-70855cb3afb8-multus-daemon-config\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705664    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705732    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg87z\" (UniqueName: \"kubernetes.io/projected/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-kube-api-access-cg87z\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:39.705821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705787    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.706321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705841    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.706321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705873    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-hostroot\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.706321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705899    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.706321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705932    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7cp\" (UniqueName: \"kubernetes.io/projected/7d027727-5c89-44a2-b1b8-20661fd13909-kube-api-access-2r7cp\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:42:39.706321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.705969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q9f\" (UniqueName: \"kubernetes.io/projected/59071108-f1ba-4346-beab-5cb766aecede-kube-api-access-29q9f\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:39.706321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.706304    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.706764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.706378    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.706764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.706434    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-kubelet-config\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.706764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.706525    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-serviceca\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:42:39.706764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.706602    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.706764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.706714    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.707104 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.706888    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.708842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.707230    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33045ca4-6bda-4fdc-9d44-f231e4161fae-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:39.710911 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.710756    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-etc-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.710911 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.710806    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-key\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:39.710911 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.710841    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.710911 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.710889    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.711141 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.710918    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7469affe-9d00-44b3-a833-4ca25b86da5c-agent-certs\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:39.711141 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.710978    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.711247 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711152    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-key\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:39.711247 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711183    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.711247 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711211    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-tmp\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.711365 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711252    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7469affe-9d00-44b3-a833-4ca25b86da5c-agent-certs\") pod \"konnectivity-agent-skbfn\" (UID: \"7469affe-9d00-44b3-a833-4ca25b86da5c\") " pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:39.711365 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711272    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.711365 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711315    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81da025c-1894-43c8-b83a-0b80409656ac-tmp\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.711365 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711353    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.711553 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711399    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.711553 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711405    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.711553 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711521    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-tmp\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.711680 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711558    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-systemd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.711680 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711591    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-cabundle\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:39.711680 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711622    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dac87d-7a94-43b5-ae69-38f6975388ed-serving-cert\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.711680 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711674    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/927717eb-ca5c-4f65-8a7a-bdc83054080c-host-slash\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:42:39.711842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711712    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysconfig\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.711842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711720    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.711842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711746    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdscw\" (UniqueName: \"kubernetes.io/projected/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a-kube-api-access-bdscw\") pod \"migrator-66df5d496b-q74kq\" (UID: \"bb4302e4-1ec4-4ce1-a136-a91a5d1d306a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"
Oct 30 00:42:39.711842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711776    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-tmp\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.711842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711788    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtt4\" (UniqueName: \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.711842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711789    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dac87d-7a94-43b5-ae69-38f6975388ed-serving-cert\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.711842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711835    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-ovn\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.712110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711897    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkbr\" (UniqueName: \"kubernetes.io/projected/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-api-access-whkbr\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.712110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711922    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.712110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711944    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-registration-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.712110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711965    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.712110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.711983    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33045ca4-6bda-4fdc-9d44-f231e4161fae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:39.712110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.712004    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gql2j\" (UniqueName: \"kubernetes.io/projected/0240e78b-bcf9-40a0-8707-0f7976d91005-kube-api-access-gql2j\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.712110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.712053    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0532776-f74d-4d8a-832a-6719d3fec3c2-signing-cabundle\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:39.712440 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.712292    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33045ca4-6bda-4fdc-9d44-f231e4161fae-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:39.712440 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.712306    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.712440 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.712394    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.717351 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.717331    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.717440 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.717350    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9nl\" (UniqueName: \"kubernetes.io/projected/4254ef03-935f-47d4-948a-70855cb3afb8-kube-api-access-4l9nl\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.717552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.717534    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjfv\" (UniqueName: \"kubernetes.io/projected/db543397-1d27-4098-a794-7a57aaa7904b-kube-api-access-ctjfv\") pod \"service-ca-operator-544d749c8c-46g6f\" (UID: \"db543397-1d27-4098-a794-7a57aaa7904b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f"
Oct 30 00:42:39.717831 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.717815    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ncq\" (UniqueName: \"kubernetes.io/projected/79ef76c0-8e88-4b4a-8d6a-83da18b39d78-kube-api-access-55ncq\") pod \"kas-connection-checker-685895c7c9-xf7xr\" (UID: \"79ef76c0-8e88-4b4a-8d6a-83da18b39d78\") " pod="kube-system/kas-connection-checker-685895c7c9-xf7xr"
Oct 30 00:42:39.718126 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.718111    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpk5j\" (UniqueName: \"kubernetes.io/projected/d56b6c9e-b741-42c5-90d7-8c8c641064fa-kube-api-access-xpk5j\") pod \"migrator-66df5d496b-2kqhm\" (UID: \"d56b6c9e-b741-42c5-90d7-8c8c641064fa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2kqhm"
Oct 30 00:42:39.719137 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.719115    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9qt\" (UniqueName: \"kubernetes.io/projected/33045ca4-6bda-4fdc-9d44-f231e4161fae-kube-api-access-sl9qt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-smb5r\" (UID: \"33045ca4-6bda-4fdc-9d44-f231e4161fae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r"
Oct 30 00:42:39.719244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.719158    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.719244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.719178    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp67x\" (UniqueName: \"kubernetes.io/projected/49fcf677-f503-492a-90e8-c95e329ee6f6-kube-api-access-wp67x\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.719889 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.719865    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qq9\" (UniqueName: \"kubernetes.io/projected/56ea1b39-0a84-4b31-a200-0c1b68cfc927-kube-api-access-w7qq9\") pod \"network-check-source-6f78d59f99-fkqq5\" (UID: \"56ea1b39-0a84-4b31-a200-0c1b68cfc927\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fkqq5"
Oct 30 00:42:39.720002 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.719981    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnf6\" (UniqueName: \"kubernetes.io/projected/2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405-kube-api-access-4cnf6\") pod \"kas-connection-checker-685895c7c9-ss447\" (UID: \"2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405\") " pod="kube-system/kas-connection-checker-685895c7c9-ss447"
Oct 30 00:42:39.720083 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.720067    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgzs\" (UniqueName: \"kubernetes.io/projected/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-kube-api-access-vdgzs\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.720130 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.720072    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pw8p\" (UniqueName: \"kubernetes.io/projected/2deae516-c4e0-4377-98ee-084581c4b027-kube-api-access-5pw8p\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:39.720222 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.720204    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzt92\" (UniqueName: \"kubernetes.io/projected/44aeb453-223b-4cfa-a838-6798b6b5aa1c-kube-api-access-fzt92\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.723842    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tx8\" (UniqueName: \"kubernetes.io/projected/b9dac87d-7a94-43b5-ae69-38f6975388ed-kube-api-access-d5tx8\") pod \"console-operator-7cd56b8dbc-5phkz\" (UID: \"b9dac87d-7a94-43b5-ae69-38f6975388ed\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724138    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg87z\" (UniqueName: \"kubernetes.io/projected/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-kube-api-access-cg87z\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724192    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2snrg\" (UniqueName: \"kubernetes.io/projected/e0532776-f74d-4d8a-832a-6719d3fec3c2-kube-api-access-2snrg\") pod \"service-ca-5fffc97bdf-q48h7\" (UID: \"e0532776-f74d-4d8a-832a-6719d3fec3c2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q48h7"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724221    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsphz\" (UniqueName: \"kubernetes.io/projected/f69dcc85-0b97-43eb-8674-0ec1415c8dfb-kube-api-access-rsphz\") pod \"network-check-target-77hhk\" (UID: \"f69dcc85-0b97-43eb-8674-0ec1415c8dfb\") " pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724318    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd285\" (UniqueName: \"kubernetes.io/projected/6a4543fe-837e-4480-b419-974b723e4475-kube-api-access-fd285\") pod \"kas-connection-checker-685895c7c9-rrv72\" (UID: \"6a4543fe-837e-4480-b419-974b723e4475\") " pod="kube-system/kas-connection-checker-685895c7c9-rrv72"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724367    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6b25\" (UniqueName: \"kubernetes.io/projected/927717eb-ca5c-4f65-8a7a-bdc83054080c-kube-api-access-l6b25\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724382    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgrf\" (UniqueName: \"kubernetes.io/projected/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-kube-api-access-qrgrf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724749    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfzl\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724825    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7j55\" (UniqueName: \"kubernetes.io/projected/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kube-api-access-p7j55\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724905    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt8t\" (UniqueName: \"kubernetes.io/projected/81da025c-1894-43c8-b83a-0b80409656ac-kube-api-access-5dt8t\") pod \"insights-operator-7c4986d449-v9wjf\" (UID: \"81da025c-1894-43c8-b83a-0b80409656ac\") " pod="openshift-insights/insights-operator-7c4986d449-v9wjf"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724916    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8tx7\" (UniqueName: \"kubernetes.io/projected/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-kube-api-access-s8tx7\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.724995    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7cp\" (UniqueName: \"kubernetes.io/projected/7d027727-5c89-44a2-b1b8-20661fd13909-kube-api-access-2r7cp\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:42:39.725036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.725005    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z546g\" (UniqueName: \"kubernetes.io/projected/9cec9dfe-c016-4228-be90-4724120f7260-kube-api-access-z546g\") pod \"volume-data-source-validator-58c9f5f46d-56c4z\" (UID: \"9cec9dfe-c016-4228-be90-4724120f7260\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-56c4z"
Oct 30 00:42:39.725432 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.725230    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62hj\" (UniqueName: \"kubernetes.io/projected/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-kube-api-access-p62hj\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.725432 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.725270    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlj97\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.725593 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.725578    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q9f\" (UniqueName: \"kubernetes.io/projected/59071108-f1ba-4346-beab-5cb766aecede-kube-api-access-29q9f\") pod \"ingress-canary-26wcv\" (UID: \"59071108-f1ba-4346-beab-5cb766aecede\") " pod="openshift-ingress-canary/ingress-canary-26wcv"
Oct 30 00:42:39.727169 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.727154    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdscw\" (UniqueName: \"kubernetes.io/projected/bb4302e4-1ec4-4ce1-a136-a91a5d1d306a-kube-api-access-bdscw\") pod \"migrator-66df5d496b-q74kq\" (UID: \"bb4302e4-1ec4-4ce1-a136-a91a5d1d306a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q74kq"
Oct 30 00:42:39.728856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.728838    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkbr\" (UniqueName: \"kubernetes.io/projected/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-api-access-whkbr\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.729058 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.729037    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gql2j\" (UniqueName: \"kubernetes.io/projected/0240e78b-bcf9-40a0-8707-0f7976d91005-kube-api-access-gql2j\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.729189 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.729176    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtt4\" (UniqueName: \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\") pod \"periodic-gathering-wmj68-h5vbk\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") " pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:42:39.813416 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813342    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.813416 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813377    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:39.813416 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813404    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-systemd\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813426    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-lib-modules\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813449    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-multus\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813499    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813515    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-log-socket\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813527    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-systemd\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.813561    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813569    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-multus\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.813580    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813598    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-lib-modules\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813616    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-log-socket\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813537    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813643    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.813653    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.313636086 +0000 UTC m=+1.861326560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:39.813711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813697    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c849f30-20d0-4135-a2b2-e5d2daef76fa-crio-socket\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813724    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-kubelet\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813747    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-multus-certs\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813773    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-bin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813792    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c849f30-20d0-4135-a2b2-e5d2daef76fa-crio-socket\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813800    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-multus-certs\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813800    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-kubernetes\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813830    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-conf-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813845    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-kubernetes\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813850    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-etc-kubernetes\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813794    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-kubelet\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813881    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813896    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-etc-kubernetes\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813899    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-conf-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813903    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813927    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813883    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-var-lib-cni-bin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813948    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-hostroot\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813990    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-hostroot\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.813991    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814013    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814020    5104 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814042    5104 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814043    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-kubelet-config\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814083    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-kubelet-config\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814100    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert podName:85585df9-d495-4d44-bd82-398d44b76cf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.314080774 +0000 UTC m=+1.861771270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s248j" (UID: "85585df9-d495-4d44-bd82-398d44b76cf8") : secret "networking-console-plugin-cert" not found
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814121    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.31411049 +0000 UTC m=+1.861800980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814118    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-etc-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814141    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814149    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814144    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-etc-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814175    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814193    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814218    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-systemd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814196    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814243    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/927717eb-ca5c-4f65-8a7a-bdc83054080c-host-slash\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814259    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814271    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814284    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814302    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/927717eb-ca5c-4f65-8a7a-bdc83054080c-host-slash\") pod \"iptables-alerter-6ns76\" (UID: \"927717eb-ca5c-4f65-8a7a-bdc83054080c\") " pod="openshift-network-operator/iptables-alerter-6ns76"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814319    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.314308538 +0000 UTC m=+1.861999054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814333    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-systemd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814351    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysconfig\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814368    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-ovn\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.814338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814374    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysconfig\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-registration-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814402    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-run\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814423    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-system-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814429    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-ovn\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814435    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-registration-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814438    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-netns\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814493    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-run-netns\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814506    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-netd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814524    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-run\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814527    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-system-cni-dir\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814534    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d027727-5c89-44a2-b1b8-20661fd13909-hosts-file\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814561    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-slash\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814567    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d027727-5c89-44a2-b1b8-20661fd13909-hosts-file\") pod \"node-resolver-rxj86\" (UID: \"7d027727-5c89-44a2-b1b8-20661fd13909\") " pod="openshift-dns/node-resolver-rxj86"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814566    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-netd\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814587    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-var-lib-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814600    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-slash\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814610    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814640    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-device-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814661    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-var-lib-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814675    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cnibin\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814703    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-cnibin\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814709    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-device-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814725    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814746    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814761    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-system-cni-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814780    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-systemd-units\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814804    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8d8e7da-34ee-466e-bf3d-48a8a56b722f-system-cni-dir\") pod \"multus-additional-cni-plugins-b6zxp\" (UID: \"a8d8e7da-34ee-466e-bf3d-48a8a56b722f\") " pod="openshift-multus/multus-additional-cni-plugins-b6zxp"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814818    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-systemd-units\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814852    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-node-log\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814867    5104 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814875    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-host\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814879    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-node-log\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814898    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814922    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-host\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814926    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.314911716 +0000 UTC m=+1.862602205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : secret "router-metrics-certs-default" not found
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814961    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-cnibin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.814962    5104 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.814989    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.314977365 +0000 UTC m=+1.862667839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815002    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-conf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815010    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-cnibin\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815024    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-host\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815048    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-kubelet\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815061    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6-host\") pod \"node-ca-r6gln\" (UID: \"e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6\") " pod="openshift-image-registry/node-ca-r6gln"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815070    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-bin\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815097    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815111    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-kubelet\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815123    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-modprobe-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815134    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-cni-bin\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815152    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815136    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-sysctl-conf\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815177    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-sys\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815204    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815227    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-etc-modprobe-d\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815231    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-run-openvswitch\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815231    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-sys-fs\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815278    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-sys-fs\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815285    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-sys\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815290    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815324    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085c11e9-3ae8-4a1d-abe7-edc7dcad9804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tntr4\" (UID: \"085c11e9-3ae8-4a1d-abe7-edc7dcad9804\") " pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.815332    5104 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815326    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-netns\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815351    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-netns\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815366    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-dbus\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.815377    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.315363373 +0000 UTC m=+1.863053861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815411    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-var-lib-kubelet\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815413    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3bb0053-40b7-49df-9490-192e78a9b720-dbus\") pod \"global-pull-secret-syncer-nk4k6\" (UID: \"b3bb0053-40b7-49df-9490-192e78a9b720\") " pod="kube-system/global-pull-secret-syncer-nk4k6"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815438    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-socket-dir-parent\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815452    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ee8ed6-aae3-42a9-adfe-3ecbfd348141-var-lib-kubelet\") pod \"tuned-ztnp7\" (UID: \"d7ee8ed6-aae3-42a9-adfe-3ecbfd348141\") " pod="openshift-cluster-node-tuning-operator/tuned-ztnp7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815483    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-k8s-cni-cncf-io\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815526    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-socket-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815530    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-host-run-k8s-cni-cncf-io\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815556    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815560    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4254ef03-935f-47d4-948a-70855cb3afb8-multus-socket-dir-parent\") pod \"multus-tcqrb\" (UID: \"4254ef03-935f-47d4-948a-70855cb3afb8\") " pod="openshift-multus/multus-tcqrb"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.815575    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e082e1-ce44-4deb-8036-89d2ae8dcc46-socket-dir\") pod \"aws-ebs-csi-driver-node-4qck7\" (UID: \"a8e082e1-ce44-4deb-8036-89d2ae8dcc46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qck7"
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.815675    5104 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:39.816272 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:39.815738    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls podName:2deae516-c4e0-4377-98ee-084581c4b027 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:40.315723461 +0000 UTC m=+1.863413953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vfg9g" (UID: "2deae516-c4e0-4377-98ee-084581c4b027") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:42:39.818728 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.817338    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49fcf677-f503-492a-90e8-c95e329ee6f6-metrics-tls\") pod \"dns-default-dkpn9\" (UID: \"49fcf677-f503-492a-90e8-c95e329ee6f6\") " pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:39.818728 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.817494    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce14d2cc-a9e8-49af-be37-9b8e6a433c65-metrics-certs\") pod \"network-metrics-daemon-2q7g6\" (UID: \"ce14d2cc-a9e8-49af-be37-9b8e6a433c65\") " pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:39.818728 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.817511    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0240e78b-bcf9-40a0-8707-0f7976d91005-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n7f4l\" (UID: \"0240e78b-bcf9-40a0-8707-0f7976d91005\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.871216 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.871182    5104 scope.go:122] "RemoveContainer" containerID="f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14"
Oct 30 00:42:39.874309 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.874288    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"
Oct 30 00:42:39.877663 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.877640    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2q7g6"
Oct 30 00:42:39.882266 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:39.882249    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:40.180621 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.180592    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l"]
Oct 30 00:42:40.188875 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.188848    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2q7g6"]
Oct 30 00:42:40.201609 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:40.201577    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce14d2cc_a9e8_49af_be37_9b8e6a433c65.slice/crio-7e008d74360966f5e13d472df7445f165c3697320b109dcae84d78d5b356d9cb WatchSource:0}: Error finding container 7e008d74360966f5e13d472df7445f165c3697320b109dcae84d78d5b356d9cb: Status 404 returned error can't find the container with id 7e008d74360966f5e13d472df7445f165c3697320b109dcae84d78d5b356d9cb
Oct 30 00:42:40.203652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.203629    5104 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:42:40.211187 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.211161    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dkpn9"]
Oct 30 00:42:40.214614 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:40.214584    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fcf677_f503_492a_90e8_c95e329ee6f6.slice/crio-6d95987d2192484346eb7f89bba5e24f48d873ee9be7d1508b960ddb112799c5 WatchSource:0}: Error finding container 6d95987d2192484346eb7f89bba5e24f48d873ee9be7d1508b960ddb112799c5: Status 404 returned error can't find the container with id 6d95987d2192484346eb7f89bba5e24f48d873ee9be7d1508b960ddb112799c5
Oct 30 00:42:40.320659 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.320622    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:40.320856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.320670    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:40.320856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.320706    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:40.320856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.320761    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:40.320856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.320814    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:40.320856 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320780    5104 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:42:40.320856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.320854    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320863    5104 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320882    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert podName:85585df9-d495-4d44-bd82-398d44b76cf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.320867421 +0000 UTC m=+2.868557925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s248j" (UID: "85585df9-d495-4d44-bd82-398d44b76cf8") : secret "networking-console-plugin-cert" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320864    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320912    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320817    5104 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320914    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.320902265 +0000 UTC m=+2.868592738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : secret "router-metrics-certs-default" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320939    5104 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320949    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.320941105 +0000 UTC m=+2.868631580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320964    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.320957567 +0000 UTC m=+2.868648040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320980    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.320971703 +0000 UTC m=+2.868662194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.320997    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.320991463 +0000 UTC m=+2.868681936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.321026    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:40.321129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.321052    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:40.321686 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.321140    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:40.321686 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.321150    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:40.321686 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.321178    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.321169841 +0000 UTC m=+2.868860315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:40.323447 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.323425    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2deae516-c4e0-4377-98ee-084581c4b027-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vfg9g\" (UID: \"2deae516-c4e0-4377-98ee-084581c4b027\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:40.430362 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.430330    5104 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:40.470356 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.470282    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"
Oct 30 00:42:40.638497 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.638441    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2q7g6" event={"ID":"ce14d2cc-a9e8-49af-be37-9b8e6a433c65","Type":"ContainerStarted","Data":"7e008d74360966f5e13d472df7445f165c3697320b109dcae84d78d5b356d9cb"}
Oct 30 00:42:40.640465 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.640427    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:42:40.640901 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.640879    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/1.log"
Oct 30 00:42:40.640994 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.640920    5104 generic.go:356] "Generic (PLEG): container finished" podID="b9dac87d-7a94-43b5-ae69-38f6975388ed" containerID="e21973d4f648780be77500d7191d4a3efe1c5aa4ff25a4a5b3de8b4323ebea3e" exitCode=255
Oct 30 00:42:40.641069 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.641035    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" event={"ID":"b9dac87d-7a94-43b5-ae69-38f6975388ed","Type":"ContainerDied","Data":"e21973d4f648780be77500d7191d4a3efe1c5aa4ff25a4a5b3de8b4323ebea3e"}
Oct 30 00:42:40.641193 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.641112    5104 scope.go:122] "RemoveContainer" containerID="f9c35f734e38240d7040d4517aed8c0690360ad8742e359a94ba4915aef0aa14"
Oct 30 00:42:40.641379 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.641362    5104 scope.go:122] "RemoveContainer" containerID="e21973d4f648780be77500d7191d4a3efe1c5aa4ff25a4a5b3de8b4323ebea3e"
Oct 30 00:42:40.641645 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:40.641619    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5phkz_openshift-console-operator(b9dac87d-7a94-43b5-ae69-38f6975388ed)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" podUID="b9dac87d-7a94-43b5-ae69-38f6975388ed"
Oct 30 00:42:40.642870 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.642841    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dkpn9" event={"ID":"49fcf677-f503-492a-90e8-c95e329ee6f6","Type":"ContainerStarted","Data":"6d95987d2192484346eb7f89bba5e24f48d873ee9be7d1508b960ddb112799c5"}
Oct 30 00:42:40.644085 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.644061    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l" event={"ID":"0240e78b-bcf9-40a0-8707-0f7976d91005","Type":"ContainerStarted","Data":"832d9f8b49a4ac1f0d45b6c72e4f2a6299317382f6fc12971b7812c97d9b0d2f"}
Oct 30 00:42:40.760027 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:40.759832    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g"]
Oct 30 00:42:40.765535 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:40.765497    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deae516_c4e0_4377_98ee_084581c4b027.slice/crio-9812163dda981485f34c053eef8b58f2560443c2a26d29f247c0b89e4aa1fcfc WatchSource:0}: Error finding container 9812163dda981485f34c053eef8b58f2560443c2a26d29f247c0b89e4aa1fcfc: Status 404 returned error can't find the container with id 9812163dda981485f34c053eef8b58f2560443c2a26d29f247c0b89e4aa1fcfc
Oct 30 00:42:41.333541 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.333490    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.333570    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.333619    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.333646    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333655    5104 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.333683    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333725    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:43.33370486 +0000 UTC m=+4.881395353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333760    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333761    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.333777    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333786    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.333816    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:41.333841 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333843    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:43.333825789 +0000 UTC m=+4.881516280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:41.334645 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333887    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:43.333876803 +0000 UTC m=+4.881567281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:41.334645 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333773    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:41.334645 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.333940    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:43.333928158 +0000 UTC m=+4.881618646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:41.334645 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.334274    5104 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:41.334645 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.334326    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:43.334309559 +0000 UTC m=+4.882000050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:41.342451 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.342414    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44aeb453-223b-4cfa-a838-6798b6b5aa1c-metrics-certs\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:41.347434 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.347360    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85585df9-d495-4d44-bd82-398d44b76cf8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s248j\" (UID: \"85585df9-d495-4d44-bd82-398d44b76cf8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:41.376515 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.376485    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j"
Oct 30 00:42:41.650768 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.650679    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2q7g6" event={"ID":"ce14d2cc-a9e8-49af-be37-9b8e6a433c65","Type":"ContainerStarted","Data":"c2597ef623b90ea1e3fabb9078ff10489040ae09ca8d920c9f763412a5e42a52"}
Oct 30 00:42:41.652840 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.652814    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:42:41.653624 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.653605    5104 scope.go:122] "RemoveContainer" containerID="e21973d4f648780be77500d7191d4a3efe1c5aa4ff25a4a5b3de8b4323ebea3e"
Oct 30 00:42:41.653827 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:41.653802    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5phkz_openshift-console-operator(b9dac87d-7a94-43b5-ae69-38f6975388ed)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" podUID="b9dac87d-7a94-43b5-ae69-38f6975388ed"
Oct 30 00:42:41.659922 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.659059    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g" event={"ID":"2deae516-c4e0-4377-98ee-084581c4b027","Type":"ContainerStarted","Data":"9812163dda981485f34c053eef8b58f2560443c2a26d29f247c0b89e4aa1fcfc"}
Oct 30 00:42:41.717075 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.717037    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-s248j"]
Oct 30 00:42:41.722985 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:41.722947    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85585df9_d495_4d44_bd82_398d44b76cf8.slice/crio-6a112da65ce94b7a54fc47cc10abef6f6759f276cd997c37f037a843c1afb1d6 WatchSource:0}: Error finding container 6a112da65ce94b7a54fc47cc10abef6f6759f276cd997c37f037a843c1afb1d6: Status 404 returned error can't find the container with id 6a112da65ce94b7a54fc47cc10abef6f6759f276cd997c37f037a843c1afb1d6
Oct 30 00:42:41.971875 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:41.971847    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:42.666881 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:42.666853    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l" event={"ID":"0240e78b-bcf9-40a0-8707-0f7976d91005","Type":"ContainerStarted","Data":"d6a8649f1ff72f0b1b54ccd4c745b7c4aea07dbb7cca22ace5b5ee07d6a38c45"}
Oct 30 00:42:42.667281 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:42.666908    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n7f4l" event={"ID":"0240e78b-bcf9-40a0-8707-0f7976d91005","Type":"ContainerStarted","Data":"47824de6cd1b23836ea5f158eacf46a12866248cfb73be5d106c2bb4b07a8dd7"}
Oct 30 00:42:42.675641 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:42.675582    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2q7g6" event={"ID":"ce14d2cc-a9e8-49af-be37-9b8e6a433c65","Type":"ContainerStarted","Data":"4ced2f8dea96430569b2c351f5a0f8c588057f3314fa58203349033c3df6f029"}
Oct 30 00:42:42.677761 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:42.677732    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j" event={"ID":"85585df9-d495-4d44-bd82-398d44b76cf8","Type":"ContainerStarted","Data":"6a112da65ce94b7a54fc47cc10abef6f6759f276cd997c37f037a843c1afb1d6"}
Oct 30 00:42:43.356328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.356230    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:43.356328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.356293    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.356352    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.356395    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356354    5104 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.356439    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356418    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356512    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7488dfd947-qp2j9: secret "image-registry-tls" not found
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356495    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:47.356479612 +0000 UTC m=+8.904170101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : configmap "kube-rbac-proxy" not found
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356555    5104 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356567    5104 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54566cf8bb-wxl8f: secret "image-registry-tls" not found
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356574    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle podName:44aeb453-223b-4cfa-a838-6798b6b5aa1c nodeName:}" failed. No retries permitted until 2026-10-30 00:42:47.356556699 +0000 UTC m=+8.904247180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle") pod "router-default-75647bc475-s8spf" (UID: "44aeb453-223b-4cfa-a838-6798b6b5aa1c") : configmap references non-existent config key: service-ca.crt
Oct 30 00:42:43.356587 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356595    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls podName:4e4d12d8-810b-45ab-8dcb-1293bc89bf40 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:47.356584874 +0000 UTC m=+8.904275353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls") pod "image-registry-7488dfd947-qp2j9" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40") : secret "image-registry-tls" not found
Oct 30 00:42:43.357122 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356614    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls podName:2296d29f-a76e-4a37-bf06-34a77068f571 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:47.356605186 +0000 UTC m=+8.904295664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls") pod "image-registry-54566cf8bb-wxl8f" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571") : secret "image-registry-tls" not found
Oct 30 00:42:43.357122 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356652    5104 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:43.357122 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:43.356687    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:47.356674178 +0000 UTC m=+8.904364666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:43.537857 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.537816    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:42:43.682826 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.682796    5104 generic.go:356] "Generic (PLEG): container finished" podID="33045ca4-6bda-4fdc-9d44-f231e4161fae" containerID="65c04cacdad84bec094b15a101d13852ddcc6c650ea4946dc48b6405db444b19" exitCode=0
Oct 30 00:42:43.683262 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.682882    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r" event={"ID":"33045ca4-6bda-4fdc-9d44-f231e4161fae","Type":"ContainerDied","Data":"65c04cacdad84bec094b15a101d13852ddcc6c650ea4946dc48b6405db444b19"}
Oct 30 00:42:43.683262 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.683232    5104 scope.go:122] "RemoveContainer" containerID="65c04cacdad84bec094b15a101d13852ddcc6c650ea4946dc48b6405db444b19"
Oct 30 00:42:43.684800 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.684734    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-s248j" event={"ID":"85585df9-d495-4d44-bd82-398d44b76cf8","Type":"ContainerStarted","Data":"6749e08267bcfdc75f474f546b0a9d5cdf47b90759988478e78d944add9e9d9f"}
Oct 30 00:42:43.686336 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.686314    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vfg9g" event={"ID":"2deae516-c4e0-4377-98ee-084581c4b027","Type":"ContainerStarted","Data":"97353b7642bc19dac677f35bad2844eaeb64db3f8b7fede0daad4890ad490619"}
Oct 30 00:42:43.688336 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.688314    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dkpn9" event={"ID":"49fcf677-f503-492a-90e8-c95e329ee6f6","Type":"ContainerStarted","Data":"1f13b12fc7b51e0b0225ac10e5e8e21349038eaeb6a4a16d9225dfe95b22dbea"}
Oct 30 00:42:43.688443 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.688356    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dkpn9" event={"ID":"49fcf677-f503-492a-90e8-c95e329ee6f6","Type":"ContainerStarted","Data":"ce87ab75b74c26580c99bd9934de9376c9d630cd5a49ca20973cf4bfb6dcc2b6"}
Oct 30 00:42:43.688443 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:43.688415    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:44.694654 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:44.694620    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-smb5r" event={"ID":"33045ca4-6bda-4fdc-9d44-f231e4161fae","Type":"ContainerStarted","Data":"7b1c074d6acdb628c938a12eaa8005c97a929425e9741c9b260d212a1367bf3d"}
Oct 30 00:42:45.614110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:45.614080    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:42:45.614351 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:45.614337    5104 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:42:46.300269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:46.300229    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:46.300716 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:46.300640    5104 scope.go:122] "RemoveContainer" containerID="e21973d4f648780be77500d7191d4a3efe1c5aa4ff25a4a5b3de8b4323ebea3e"
Oct 30 00:42:46.300826 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:46.300806    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5phkz_openshift-console-operator(b9dac87d-7a94-43b5-ae69-38f6975388ed)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" podUID="b9dac87d-7a94-43b5-ae69-38f6975388ed"
Oct 30 00:42:47.399381 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.399340    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:47.399381 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.399387    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:47.399854 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.399428    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:47.399854 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.399484    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:47.399854 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.399561    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:47.399854 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:47.399592    5104 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:47.399854 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:42:47.399683    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls podName:1c849f30-20d0-4135-a2b2-e5d2daef76fa nodeName:}" failed. No retries permitted until 2026-10-30 00:42:55.399662681 +0000 UTC m=+16.947353157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmwj7" (UID: "1c849f30-20d0-4135-a2b2-e5d2daef76fa") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:42:47.400082 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.400047    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44aeb453-223b-4cfa-a838-6798b6b5aa1c-service-ca-bundle\") pod \"router-default-75647bc475-s8spf\" (UID: \"44aeb453-223b-4cfa-a838-6798b6b5aa1c\") " pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:47.400082 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.400076    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c849f30-20d0-4135-a2b2-e5d2daef76fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:47.402029 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.402007    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"image-registry-54566cf8bb-wxl8f\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") " pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:47.402119 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.402048    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"image-registry-7488dfd947-qp2j9\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") " pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:47.675617 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.675583    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:47.675790 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.675692    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:47.683563 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:47.682000    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:48.013612 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.013525    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54566cf8bb-wxl8f"]
Oct 30 00:42:48.016561 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.016538    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7488dfd947-qp2j9"]
Oct 30 00:42:48.018035 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:48.018012    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2296d29f_a76e_4a37_bf06_34a77068f571.slice/crio-5939ca7cf89d853b8b51b5784080274a5354680cca4dc9b2d7cc79341330dc08 WatchSource:0}: Error finding container 5939ca7cf89d853b8b51b5784080274a5354680cca4dc9b2d7cc79341330dc08: Status 404 returned error can't find the container with id 5939ca7cf89d853b8b51b5784080274a5354680cca4dc9b2d7cc79341330dc08
Oct 30 00:42:48.019852 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:48.019832    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4d12d8_810b_45ab_8dcb_1293bc89bf40.slice/crio-8d0a68a3698f5369ebb8461c7d8f303175f11ebb55e9b8d0f64b533e24190fe8 WatchSource:0}: Error finding container 8d0a68a3698f5369ebb8461c7d8f303175f11ebb55e9b8d0f64b533e24190fe8: Status 404 returned error can't find the container with id 8d0a68a3698f5369ebb8461c7d8f303175f11ebb55e9b8d0f64b533e24190fe8
Oct 30 00:42:48.024588 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.024565    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75647bc475-s8spf"]
Oct 30 00:42:48.027849 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:48.027667    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aeb453_223b_4cfa_a838_6798b6b5aa1c.slice/crio-7a4c56a02da1f743e2996221632f4eb5342dfb6b77bdf90f34736b4e74cdcbb6 WatchSource:0}: Error finding container 7a4c56a02da1f743e2996221632f4eb5342dfb6b77bdf90f34736b4e74cdcbb6: Status 404 returned error can't find the container with id 7a4c56a02da1f743e2996221632f4eb5342dfb6b77bdf90f34736b4e74cdcbb6
Oct 30 00:42:48.718943 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.718905    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f" event={"ID":"2296d29f-a76e-4a37-bf06-34a77068f571","Type":"ContainerStarted","Data":"29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf"}
Oct 30 00:42:48.719363 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.718955    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f" event={"ID":"2296d29f-a76e-4a37-bf06-34a77068f571","Type":"ContainerStarted","Data":"5939ca7cf89d853b8b51b5784080274a5354680cca4dc9b2d7cc79341330dc08"}
Oct 30 00:42:48.720509 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.720483    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75647bc475-s8spf" event={"ID":"44aeb453-223b-4cfa-a838-6798b6b5aa1c","Type":"ContainerStarted","Data":"4e8bd83af07286af87bc35fe4c2255edd8cd5910a15915471c4854bae127bc53"}
Oct 30 00:42:48.720658 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.720529    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75647bc475-s8spf" event={"ID":"44aeb453-223b-4cfa-a838-6798b6b5aa1c","Type":"ContainerStarted","Data":"7a4c56a02da1f743e2996221632f4eb5342dfb6b77bdf90f34736b4e74cdcbb6"}
Oct 30 00:42:48.721956 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.721931    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9" event={"ID":"4e4d12d8-810b-45ab-8dcb-1293bc89bf40","Type":"ContainerStarted","Data":"082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f"}
Oct 30 00:42:48.722057 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.721974    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9" event={"ID":"4e4d12d8-810b-45ab-8dcb-1293bc89bf40","Type":"ContainerStarted","Data":"8d0a68a3698f5369ebb8461c7d8f303175f11ebb55e9b8d0f64b533e24190fe8"}
Oct 30 00:42:48.770824 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:48.770776    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:42:49.676154 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:49.676116    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:49.678952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:49.678923    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:49.726017 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:49.725994    5104 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:42:49.775950 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:49.775922    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-skbfn"
Oct 30 00:42:50.431212 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:50.431168    5104 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:50.431606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:50.431591    5104 scope.go:122] "RemoveContainer" containerID="e21973d4f648780be77500d7191d4a3efe1c5aa4ff25a4a5b3de8b4323ebea3e"
Oct 30 00:42:50.732471 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:50.732371    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:42:50.732912 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:50.732467    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz" event={"ID":"b9dac87d-7a94-43b5-ae69-38f6975388ed","Type":"ContainerStarted","Data":"19c5effa78063410ed4c86399e9b2e489275c1a5922277157a900a138deb3ee3"}
Oct 30 00:42:50.732912 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:50.732505    5104 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:42:50.733107 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:50.733084    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:50.853574 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:50.853546    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5phkz"
Oct 30 00:42:53.697017 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:53.696987    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dkpn9"
Oct 30 00:42:54.422939 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:54.422907    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:42:55.471302 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:55.471258    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:55.473645 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:55.473613    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c849f30-20d0-4135-a2b2-e5d2daef76fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwj7\" (UID: \"1c849f30-20d0-4135-a2b2-e5d2daef76fa\") " pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:55.484588 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:55.484566    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lmwj7"
Oct 30 00:42:55.774912 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:55.774829    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lmwj7"]
Oct 30 00:42:56.759718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:56.759630    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwj7" event={"ID":"1c849f30-20d0-4135-a2b2-e5d2daef76fa","Type":"ContainerStarted","Data":"cd42f7ac37e36c728cd148cd6869f7861941be7e114524c22549ac5fd240132e"}
Oct 30 00:42:56.759718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:56.759696    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwj7" event={"ID":"1c849f30-20d0-4135-a2b2-e5d2daef76fa","Type":"ContainerStarted","Data":"cab6d4f4e1c88fab48e84c34c7f82636b017662f4f936bc1d3402690dfdba16b"}
Oct 30 00:42:56.759718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:56.759716    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwj7" event={"ID":"1c849f30-20d0-4135-a2b2-e5d2daef76fa","Type":"ContainerStarted","Data":"1db8e21fb01f31e8b23c0ca46bbb004603151f833f0ade997bbfc985e058610f"}
Oct 30 00:42:56.836822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:56.836785    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:56.836969 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:56.836918    5104 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:42:56.838105 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:56.838082    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75647bc475-s8spf"
Oct 30 00:42:58.769842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.769805    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwj7" event={"ID":"1c849f30-20d0-4135-a2b2-e5d2daef76fa","Type":"ContainerStarted","Data":"460ef267d8b1e11390e7661283954f9b0ebe2a1b395d883b9305cc886313fc52"}
Oct 30 00:42:58.869344 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.869302    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"]
Oct 30 00:42:58.872876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.872847    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"
Oct 30 00:42:58.876916 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.876889    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54566cf8bb-wxl8f"]
Oct 30 00:42:58.893863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.893832    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2zh82\""
Oct 30 00:42:58.893863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.893854    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 00:42:58.895716 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.895646    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"]
Oct 30 00:42:58.900791 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.900755    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7a355cec-6892-4b36-89b7-60c6b3460bf0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wfw4k\" (UID: \"7a355cec-6892-4b36-89b7-60c6b3460bf0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"
Oct 30 00:42:58.968073 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.968039    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-9ks9h"]
Oct 30 00:42:58.971705 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.971683    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:58.974373 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.974353    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bbf7cb548-clh8s"]
Oct 30 00:42:58.977568 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.977541    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:58.991491 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.991464    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 00:42:58.991685 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.991661    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 00:42:58.992747 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.992729    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9ks9h"]
Oct 30 00:42:58.993901 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.993882    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-77f52\""
Oct 30 00:42:58.999175 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:58.999154    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bbf7cb548-clh8s"]
Oct 30 00:42:59.001943 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.001916    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d581f7f9-2281-4635-a1ef-4930a5ad7026-tmp\") pod \"downloads-6dd789c8b7-9ks9h\" (UID: \"d581f7f9-2281-4635-a1ef-4930a5ad7026\") " pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:59.002046 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.001992    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzns\" (UniqueName: \"kubernetes.io/projected/d581f7f9-2281-4635-a1ef-4930a5ad7026-kube-api-access-clzns\") pod \"downloads-6dd789c8b7-9ks9h\" (UID: \"d581f7f9-2281-4635-a1ef-4930a5ad7026\") " pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:59.002138 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.002118    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7a355cec-6892-4b36-89b7-60c6b3460bf0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wfw4k\" (UID: \"7a355cec-6892-4b36-89b7-60c6b3460bf0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"
Oct 30 00:42:59.004687 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.004668    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7a355cec-6892-4b36-89b7-60c6b3460bf0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wfw4k\" (UID: \"7a355cec-6892-4b36-89b7-60c6b3460bf0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"
Oct 30 00:42:59.102730 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102639    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55c2f131-88b8-4125-b2c9-be2d4704c95b-trusted-ca\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.102730 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102677    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-bound-sa-token\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.102730 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102709    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk48b\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-kube-api-access-lk48b\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.102942 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102815    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzns\" (UniqueName: \"kubernetes.io/projected/d581f7f9-2281-4635-a1ef-4930a5ad7026-kube-api-access-clzns\") pod \"downloads-6dd789c8b7-9ks9h\" (UID: \"d581f7f9-2281-4635-a1ef-4930a5ad7026\") " pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:59.102942 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102838    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55c2f131-88b8-4125-b2c9-be2d4704c95b-installation-pull-secrets\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.102942 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102873    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55c2f131-88b8-4125-b2c9-be2d4704c95b-registry-certificates\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.102942 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102927    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55c2f131-88b8-4125-b2c9-be2d4704c95b-ca-trust-extracted\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.103065 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102977    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-registry-tls\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.103065 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.102997    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d581f7f9-2281-4635-a1ef-4930a5ad7026-tmp\") pod \"downloads-6dd789c8b7-9ks9h\" (UID: \"d581f7f9-2281-4635-a1ef-4930a5ad7026\") " pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:59.103065 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.103012    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55c2f131-88b8-4125-b2c9-be2d4704c95b-image-registry-private-configuration\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.103359 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.103343    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d581f7f9-2281-4635-a1ef-4930a5ad7026-tmp\") pod \"downloads-6dd789c8b7-9ks9h\" (UID: \"d581f7f9-2281-4635-a1ef-4930a5ad7026\") " pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:59.121701 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.121672    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzns\" (UniqueName: \"kubernetes.io/projected/d581f7f9-2281-4635-a1ef-4930a5ad7026-kube-api-access-clzns\") pod \"downloads-6dd789c8b7-9ks9h\" (UID: \"d581f7f9-2281-4635-a1ef-4930a5ad7026\") " pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:59.187097 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.187059    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"
Oct 30 00:42:59.203551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203519    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55c2f131-88b8-4125-b2c9-be2d4704c95b-image-registry-private-configuration\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.203670 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203572    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55c2f131-88b8-4125-b2c9-be2d4704c95b-trusted-ca\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.203670 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203594    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-bound-sa-token\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.203670 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203615    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk48b\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-kube-api-access-lk48b\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.203670 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203643    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55c2f131-88b8-4125-b2c9-be2d4704c95b-installation-pull-secrets\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.203896 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203795    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55c2f131-88b8-4125-b2c9-be2d4704c95b-registry-certificates\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.203896 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203870    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55c2f131-88b8-4125-b2c9-be2d4704c95b-ca-trust-extracted\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.204006 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.203968    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-registry-tls\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.204768 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.204412    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55c2f131-88b8-4125-b2c9-be2d4704c95b-ca-trust-extracted\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.204872 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.204800    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55c2f131-88b8-4125-b2c9-be2d4704c95b-registry-certificates\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.204960 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.204935    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55c2f131-88b8-4125-b2c9-be2d4704c95b-trusted-ca\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.206192 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.206173    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55c2f131-88b8-4125-b2c9-be2d4704c95b-image-registry-private-configuration\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.206292 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.206248    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55c2f131-88b8-4125-b2c9-be2d4704c95b-installation-pull-secrets\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.206813 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.206796    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-registry-tls\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.241339 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.241298    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-bound-sa-token\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.241910 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.241891    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk48b\" (UniqueName: \"kubernetes.io/projected/55c2f131-88b8-4125-b2c9-be2d4704c95b-kube-api-access-lk48b\") pod \"image-registry-7bbf7cb548-clh8s\" (UID: \"55c2f131-88b8-4125-b2c9-be2d4704c95b\") " pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.283997 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.283965    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:42:59.290357 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.290330    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.489614 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.489583    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"]
Oct 30 00:42:59.493064 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:59.493038    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a355cec_6892_4b36_89b7_60c6b3460bf0.slice/crio-257b3edbb93a1797e4776f06850c44e7c807876ecfca2da8685be5ace8cf6386 WatchSource:0}: Error finding container 257b3edbb93a1797e4776f06850c44e7c807876ecfca2da8685be5ace8cf6386: Status 404 returned error can't find the container with id 257b3edbb93a1797e4776f06850c44e7c807876ecfca2da8685be5ace8cf6386
Oct 30 00:42:59.583834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.583777    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9ks9h"]
Oct 30 00:42:59.587197 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:59.587156    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd581f7f9_2281_4635_a1ef_4930a5ad7026.slice/crio-2f3096833d52b8f6d6bc8450071d310faec8df14a5c99ec5af682a412be50475 WatchSource:0}: Error finding container 2f3096833d52b8f6d6bc8450071d310faec8df14a5c99ec5af682a412be50475: Status 404 returned error can't find the container with id 2f3096833d52b8f6d6bc8450071d310faec8df14a5c99ec5af682a412be50475
Oct 30 00:42:59.588174 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.588136    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bbf7cb548-clh8s"]
Oct 30 00:42:59.591921 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:42:59.591899    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c2f131_88b8_4125_b2c9_be2d4704c95b.slice/crio-4771e855abab26bfd5e0e45b804118aaf55c41affa2f48b1415ba44e6f5234fc WatchSource:0}: Error finding container 4771e855abab26bfd5e0e45b804118aaf55c41affa2f48b1415ba44e6f5234fc: Status 404 returned error can't find the container with id 4771e855abab26bfd5e0e45b804118aaf55c41affa2f48b1415ba44e6f5234fc
Oct 30 00:42:59.775250 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.775210    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k" event={"ID":"7a355cec-6892-4b36-89b7-60c6b3460bf0","Type":"ContainerStarted","Data":"257b3edbb93a1797e4776f06850c44e7c807876ecfca2da8685be5ace8cf6386"}
Oct 30 00:42:59.776814 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.776786    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s" event={"ID":"55c2f131-88b8-4125-b2c9-be2d4704c95b","Type":"ContainerStarted","Data":"1dc552532faff6a00ef53e15a2cc1677b6980ecff8ee751bfbb2f763ed4dbf07"}
Oct 30 00:42:59.776962 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.776833    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s" event={"ID":"55c2f131-88b8-4125-b2c9-be2d4704c95b","Type":"ContainerStarted","Data":"4771e855abab26bfd5e0e45b804118aaf55c41affa2f48b1415ba44e6f5234fc"}
Oct 30 00:42:59.776962 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.776907    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:42:59.778009 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.777987    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9ks9h" event={"ID":"d581f7f9-2281-4635-a1ef-4930a5ad7026","Type":"ContainerStarted","Data":"2f3096833d52b8f6d6bc8450071d310faec8df14a5c99ec5af682a412be50475"}
Oct 30 00:42:59.815946 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:42:59.815898    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s" podStartSLOduration=1.815883683 podStartE2EDuration="1.815883683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:59.815383605 +0000 UTC m=+21.363074110" watchObservedRunningTime="2026-10-30 00:42:59.815883683 +0000 UTC m=+21.363574179"
Oct 30 00:43:00.785517 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:00.785478    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k" event={"ID":"7a355cec-6892-4b36-89b7-60c6b3460bf0","Type":"ContainerStarted","Data":"ffdae1968440c2e3272b8d93373e7ca83f8baea6d216109e8f272fce07c67aba"}
Oct 30 00:43:00.821657 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:00.821599    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k" podStartSLOduration=1.727577184 podStartE2EDuration="2.821579915s" totalImagesPullingTime="1.094002731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:00.820954724 +0000 UTC m=+22.368645220" watchObservedRunningTime="2026-10-30 00:43:00.821579915 +0000 UTC m=+22.369270415"
Oct 30 00:43:01.790756 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:01.790719    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"
Oct 30 00:43:01.797362 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:01.797332    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k"
Oct 30 00:43:02.821256 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:02.821215    5104 patch_prober.go:29] interesting pod/image-registry-7488dfd947-qp2j9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 00:43:02.821740 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:02.821285    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9" podUID="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:43:04.081618 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:04.081586    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tntr4"
Oct 30 00:43:06.878814 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.878776    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4lnp9"]
Oct 30 00:43:06.915428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.915392    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.944369 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.944335    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:43:06.944648 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.944497    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:43:06.955813 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.955777    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:43:06.955813 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.955778    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:43:06.956045 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.955788    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rbt7f\""
Oct 30 00:43:06.979047 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979011    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-textfile\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979218 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979053    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-accelerators-collector-config\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979218 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979079    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-wtmp\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979218 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979208    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-sys\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979248    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mckqb\" (UniqueName: \"kubernetes.io/projected/276c6084-0984-48d8-886a-9ef52e789339-kube-api-access-mckqb\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979349    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979390    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/276c6084-0984-48d8-886a-9ef52e789339-metrics-client-ca\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979428    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-root\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:06.979556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:06.979476    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-tls\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.080964 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.080922    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081009    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/276c6084-0984-48d8-886a-9ef52e789339-metrics-client-ca\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081045    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-root\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081066    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-tls\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081101    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-textfile\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081120    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-accelerators-collector-config\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081151    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-wtmp\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081173    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-sys\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081608 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081191    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mckqb\" (UniqueName: \"kubernetes.io/projected/276c6084-0984-48d8-886a-9ef52e789339-kube-api-access-mckqb\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081608 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081206    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-root\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081658    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-wtmp\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081767 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081715    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/276c6084-0984-48d8-886a-9ef52e789339-sys\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.081961 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081919    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/276c6084-0984-48d8-886a-9ef52e789339-metrics-client-ca\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.082144 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.081975    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-accelerators-collector-config\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.082144 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.082058    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-textfile\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.083794 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.083769    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.084265 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.084247    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/276c6084-0984-48d8-886a-9ef52e789339-node-exporter-tls\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.100403 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.100370    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mckqb\" (UniqueName: \"kubernetes.io/projected/276c6084-0984-48d8-886a-9ef52e789339-kube-api-access-mckqb\") pod \"node-exporter-4lnp9\" (UID: \"276c6084-0984-48d8-886a-9ef52e789339\") " pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.284642 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.284605    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4lnp9"
Oct 30 00:43:07.309087 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:43:07.309051    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276c6084_0984_48d8_886a_9ef52e789339.slice/crio-deeea3b496d1f1e2b0291eda9bca5d0a4a5f87855f68ffcaab79430bce673bb9 WatchSource:0}: Error finding container deeea3b496d1f1e2b0291eda9bca5d0a4a5f87855f68ffcaab79430bce673bb9: Status 404 returned error can't find the container with id deeea3b496d1f1e2b0291eda9bca5d0a4a5f87855f68ffcaab79430bce673bb9
Oct 30 00:43:07.822211 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:07.822126    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4lnp9" event={"ID":"276c6084-0984-48d8-886a-9ef52e789339","Type":"ContainerStarted","Data":"deeea3b496d1f1e2b0291eda9bca5d0a4a5f87855f68ffcaab79430bce673bb9"}
Oct 30 00:43:08.829930 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:08.829893    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4lnp9" event={"ID":"276c6084-0984-48d8-886a-9ef52e789339","Type":"ContainerStarted","Data":"e752b6e357618981295139d8e92ab6ad8e9c7d0dfb2077112b6c328cf8461509"}
Oct 30 00:43:08.884447 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:08.884411    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:43:09.106813 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:43:09.106773    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:43:09.836978 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:09.836944    5104 generic.go:356] "Generic (PLEG): container finished" podID="276c6084-0984-48d8-886a-9ef52e789339" containerID="e752b6e357618981295139d8e92ab6ad8e9c7d0dfb2077112b6c328cf8461509" exitCode=0
Oct 30 00:43:09.837486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:09.837018    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4lnp9" event={"ID":"276c6084-0984-48d8-886a-9ef52e789339","Type":"ContainerDied","Data":"e752b6e357618981295139d8e92ab6ad8e9c7d0dfb2077112b6c328cf8461509"}
Oct 30 00:43:10.738968 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:10.738919    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:43:10.848514 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:10.848428    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4lnp9" event={"ID":"276c6084-0984-48d8-886a-9ef52e789339","Type":"ContainerStarted","Data":"7b3adfc4342f539f80e38f4088d4ce1f68b31fcf83d05f327c4404ce1dcb06b2"}
Oct 30 00:43:10.849020 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:10.848519    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4lnp9" event={"ID":"276c6084-0984-48d8-886a-9ef52e789339","Type":"ContainerStarted","Data":"fb47c28ca9c9241b268da7c285d0342132f33c18d6ea8c604edbaf9cf9957044"}
Oct 30 00:43:10.888675 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:10.888622    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4lnp9" podStartSLOduration=3.458909288 podStartE2EDuration="4.888607821s" totalImagesPullingTime="1.341528847s" totalInitContainerRuntime="88.169686ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:10.887421405 +0000 UTC m=+32.435111900" watchObservedRunningTime="2026-10-30 00:43:10.888607821 +0000 UTC m=+32.436298316"
Oct 30 00:43:13.542027 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:13.541993    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-77hhk"
Oct 30 00:43:16.879799 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:16.879759    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9ks9h" event={"ID":"d581f7f9-2281-4635-a1ef-4930a5ad7026","Type":"ContainerStarted","Data":"7f60be6b747e45995886794c3b23147f027ee4fe22a517906fff1b1319a3b6bd"}
Oct 30 00:43:16.880313 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:16.879940    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:43:16.897030 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:16.896997    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-9ks9h"
Oct 30 00:43:16.940963 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:16.940901    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-9ks9h" podStartSLOduration=2.552732355 podStartE2EDuration="18.940884572s" totalImagesPullingTime="16.388152217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:16.929195096 +0000 UTC m=+38.476885592" watchObservedRunningTime="2026-10-30 00:43:16.940884572 +0000 UTC m=+38.488575065"
Oct 30 00:43:20.791032 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:20.790939    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bbf7cb548-clh8s"
Oct 30 00:43:20.880443 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:20.880403    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7488dfd947-qp2j9"]
Oct 30 00:43:23.900001 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:23.899947    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f" podUID="2296d29f-a76e-4a37-bf06-34a77068f571" containerName="registry" containerID="cri-o://29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf" gracePeriod=30
Oct 30 00:43:24.205259 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.205223    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:43:24.346822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.346729    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.346822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.346801    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.347071 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.346841    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.347071 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.346893    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.347071 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.346966    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.347071 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.347013    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.347071 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.347054    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.347331 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.347095    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\") pod \"2296d29f-a76e-4a37-bf06-34a77068f571\" (UID: \"2296d29f-a76e-4a37-bf06-34a77068f571\") "
Oct 30 00:43:24.347831 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.347793    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:24.348262 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.348229    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:24.351570 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.351535    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:24.351686 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.351584    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "kube-api-access-rlj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:24.351756 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.351704    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:24.351829 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.351775    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:24.352628 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.352599    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:24.356813 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.356783    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted" pod "2296d29f-a76e-4a37-bf06-34a77068f571" (UID: "2296d29f-a76e-4a37-bf06-34a77068f571"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:43:24.448149 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448115    5104 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-bound-sa-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.448149 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448149    5104 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-registry-certificates\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.448353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448160    5104 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-registry-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.448353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448169    5104 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-installation-pull-secrets\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.448353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448178    5104 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2296d29f-a76e-4a37-bf06-34a77068f571-ca-trust-extracted\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.448353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448192    5104 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2296d29f-a76e-4a37-bf06-34a77068f571-trusted-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.448353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448208    5104 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2296d29f-a76e-4a37-bf06-34a77068f571-image-registry-private-configuration\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.448353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.448218    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlj97\" (UniqueName: \"kubernetes.io/projected/2296d29f-a76e-4a37-bf06-34a77068f571-kube-api-access-rlj97\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:24.922389 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.922350    5104 generic.go:356] "Generic (PLEG): container finished" podID="2296d29f-a76e-4a37-bf06-34a77068f571" containerID="29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf" exitCode=0
Oct 30 00:43:24.922907 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.922418    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f"
Oct 30 00:43:24.922907 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.922433    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f" event={"ID":"2296d29f-a76e-4a37-bf06-34a77068f571","Type":"ContainerDied","Data":"29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf"}
Oct 30 00:43:24.922907 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.922508    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54566cf8bb-wxl8f" event={"ID":"2296d29f-a76e-4a37-bf06-34a77068f571","Type":"ContainerDied","Data":"5939ca7cf89d853b8b51b5784080274a5354680cca4dc9b2d7cc79341330dc08"}
Oct 30 00:43:24.922907 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.922530    5104 scope.go:122] "RemoveContainer" containerID="29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf"
Oct 30 00:43:24.933226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.933210    5104 scope.go:122] "RemoveContainer" containerID="29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf"
Oct 30 00:43:24.933534 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:43:24.933506    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf\": container with ID starting with 29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf not found: ID does not exist" containerID="29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf"
Oct 30 00:43:24.933623 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.933544    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf"} err="failed to get container status \"29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf\": rpc error: code = NotFound desc = could not find container \"29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf\": container with ID starting with 29707b7fd9418c9e063eca9b985a2a08ec0bffa36e596f51d887047170e5b5bf not found: ID does not exist"
Oct 30 00:43:24.982302 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.982265    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54566cf8bb-wxl8f"]
Oct 30 00:43:24.993801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:24.993772    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54566cf8bb-wxl8f"]
Oct 30 00:43:26.574762 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:26.574729    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2296d29f-a76e-4a37-bf06-34a77068f571" path="/var/lib/kubelet/pods/2296d29f-a76e-4a37-bf06-34a77068f571/volumes"
Oct 30 00:43:39.103302 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:43:39.103264    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:43:45.906267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:45.906212    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9" podUID="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" containerName="registry" containerID="cri-o://082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f" gracePeriod=30
Oct 30 00:43:46.193936 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.193903    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:43:46.224198 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224163    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.224198 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224202    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.224436 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224222    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.224436 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224245    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.224436 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224275    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.224436 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224377    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.224669 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224511    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.224669 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.224562    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\") pod \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\" (UID: \"4e4d12d8-810b-45ab-8dcb-1293bc89bf40\") "
Oct 30 00:43:46.225082 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.225002    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:46.225291 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.225251    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:46.226888 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.226859    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:46.227007 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.226988    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:46.227075 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.227013    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:46.227116 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.227071    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:46.227210 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.227194    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "kube-api-access-6sfzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:46.231376 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.231354    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted" pod "4e4d12d8-810b-45ab-8dcb-1293bc89bf40" (UID: "4e4d12d8-810b-45ab-8dcb-1293bc89bf40"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:43:46.325769 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325724    5104 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-trusted-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:46.325769 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325760    5104 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-bound-sa-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:46.325769 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325775    5104 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-image-registry-private-configuration\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:46.326015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325791    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfzl\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-kube-api-access-6sfzl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:46.326015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325806    5104 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-installation-pull-secrets\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:46.326015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325818    5104 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:46.326015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325829    5104 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-ca-trust-extracted\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:46.326015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:46.325840    5104 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4d12d8-810b-45ab-8dcb-1293bc89bf40-registry-certificates\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:47.012516 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.012480    5104 generic.go:356] "Generic (PLEG): container finished" podID="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" containerID="082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f" exitCode=0
Oct 30 00:43:47.012953 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.012525    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9"
Oct 30 00:43:47.012953 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.012550    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9" event={"ID":"4e4d12d8-810b-45ab-8dcb-1293bc89bf40","Type":"ContainerDied","Data":"082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f"}
Oct 30 00:43:47.012953 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.012606    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7488dfd947-qp2j9" event={"ID":"4e4d12d8-810b-45ab-8dcb-1293bc89bf40","Type":"ContainerDied","Data":"8d0a68a3698f5369ebb8461c7d8f303175f11ebb55e9b8d0f64b533e24190fe8"}
Oct 30 00:43:47.012953 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.012628    5104 scope.go:122] "RemoveContainer" containerID="082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f"
Oct 30 00:43:47.021186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.021167    5104 scope.go:122] "RemoveContainer" containerID="082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f"
Oct 30 00:43:47.021472 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:43:47.021438    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f\": container with ID starting with 082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f not found: ID does not exist" containerID="082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f"
Oct 30 00:43:47.021523 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.021477    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f"} err="failed to get container status \"082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f\": rpc error: code = NotFound desc = could not find container \"082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f\": container with ID starting with 082bcb416e048c3ae3bd33acb3ac01e8fff131bcd1d38db840e96479c508656f not found: ID does not exist"
Oct 30 00:43:47.071423 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.071391    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7488dfd947-qp2j9"]
Oct 30 00:43:47.083033 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:47.083003    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7488dfd947-qp2j9"]
Oct 30 00:43:48.573982 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:48.573951    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" path="/var/lib/kubelet/pods/4e4d12d8-810b-45ab-8dcb-1293bc89bf40/volumes"
Oct 30 00:43:53.039421 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.039389    5104 generic.go:356] "Generic (PLEG): container finished" podID="81da025c-1894-43c8-b83a-0b80409656ac" containerID="83e26191b6364d6f227113cc9f7d4a3bd68b0520348306812882e41575bd3c16" exitCode=0
Oct 30 00:43:53.039899 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.039473    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wjf" event={"ID":"81da025c-1894-43c8-b83a-0b80409656ac","Type":"ContainerDied","Data":"83e26191b6364d6f227113cc9f7d4a3bd68b0520348306812882e41575bd3c16"}
Oct 30 00:43:53.039899 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.039839    5104 scope.go:122] "RemoveContainer" containerID="83e26191b6364d6f227113cc9f7d4a3bd68b0520348306812882e41575bd3c16"
Oct 30 00:43:53.501353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.501321    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wmj68-h5vbk"]
Oct 30 00:43:53.501576 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.501543    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-wmj68-h5vbk" podUID="15ba866d-3fba-4bf9-beec-14ecda492c3f" containerName="insights-gathering" containerID="cri-o://dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7" gracePeriod=30
Oct 30 00:43:53.685606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.685567    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wccd7-qxqqb"]
Oct 30 00:43:53.685958 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.685940    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2296d29f-a76e-4a37-bf06-34a77068f571" containerName="registry"
Oct 30 00:43:53.685958 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.685959    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2296d29f-a76e-4a37-bf06-34a77068f571" containerName="registry"
Oct 30 00:43:53.686129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.685974    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" containerName="registry"
Oct 30 00:43:53.686129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.685982    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" containerName="registry"
Oct 30 00:43:53.686129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.686047    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2296d29f-a76e-4a37-bf06-34a77068f571" containerName="registry"
Oct 30 00:43:53.686129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.686059    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4d12d8-810b-45ab-8dcb-1293bc89bf40" containerName="registry"
Oct 30 00:43:53.708022 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.707697    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wccd7-qxqqb"]
Oct 30 00:43:53.710320 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.710258    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.788518 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.788489    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68b482a7-e9e0-4787-96f3-8949ab725a4d-archives-path\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.788652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.788524    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvjn\" (UniqueName: \"kubernetes.io/projected/68b482a7-e9e0-4787-96f3-8949ab725a4d-kube-api-access-prvjn\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.788652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.788544    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-service-ca-bundle\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.788750 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.788732    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-trusted-ca-bundle\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.804730 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.804701    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:43:53.889723 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.889684    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\") pod \"15ba866d-3fba-4bf9-beec-14ecda492c3f\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") "
Oct 30 00:43:53.889896 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.889741    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\" (UniqueName: \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\") pod \"15ba866d-3fba-4bf9-beec-14ecda492c3f\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") "
Oct 30 00:43:53.889896 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.889779    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\") pod \"15ba866d-3fba-4bf9-beec-14ecda492c3f\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") "
Oct 30 00:43:53.889896 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.889800    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\") pod \"15ba866d-3fba-4bf9-beec-14ecda492c3f\" (UID: \"15ba866d-3fba-4bf9-beec-14ecda492c3f\") "
Oct 30 00:43:53.890016 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.889944    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-trusted-ca-bundle\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.890059 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890017    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68b482a7-e9e0-4787-96f3-8949ab725a4d-archives-path\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.890059 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890044    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvjn\" (UniqueName: \"kubernetes.io/projected/68b482a7-e9e0-4787-96f3-8949ab725a4d-kube-api-access-prvjn\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.890150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890079    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-service-ca-bundle\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.890207 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890148    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle" pod "15ba866d-3fba-4bf9-beec-14ecda492c3f" (UID: "15ba866d-3fba-4bf9-beec-14ecda492c3f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:53.890207 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890168    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path" pod "15ba866d-3fba-4bf9-beec-14ecda492c3f" (UID: "15ba866d-3fba-4bf9-beec-14ecda492c3f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:43:53.890207 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890182    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle" pod "15ba866d-3fba-4bf9-beec-14ecda492c3f" (UID: "15ba866d-3fba-4bf9-beec-14ecda492c3f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:53.890554 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890523    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68b482a7-e9e0-4787-96f3-8949ab725a4d-archives-path\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.890763 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890745    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-service-ca-bundle\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.890939 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.890920    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-trusted-ca-bundle\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.892188 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.892171    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4" pod "15ba866d-3fba-4bf9-beec-14ecda492c3f" (UID: "15ba866d-3fba-4bf9-beec-14ecda492c3f"). InnerVolumeSpecName "kube-api-access-pxtt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:53.908295 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.908271    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvjn\" (UniqueName: \"kubernetes.io/projected/68b482a7-e9e0-4787-96f3-8949ab725a4d-kube-api-access-prvjn\") pod \"periodic-gathering-wccd7-qxqqb\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") " pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:53.991202 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.991164    5104 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/15ba866d-3fba-4bf9-beec-14ecda492c3f-archives-path\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:53.991202 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.991194    5104 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-service-ca-bundle\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:53.991202 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.991204    5104 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15ba866d-3fba-4bf9-beec-14ecda492c3f-trusted-ca-bundle\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:53.991202 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:53.991214    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtt4\" (UniqueName: \"kubernetes.io/projected/15ba866d-3fba-4bf9-beec-14ecda492c3f-kube-api-access-pxtt4\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:43:54.044857 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.044765    5104 generic.go:356] "Generic (PLEG): container finished" podID="15ba866d-3fba-4bf9-beec-14ecda492c3f" containerID="dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7" exitCode=2
Oct 30 00:43:54.044857 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.044811    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wmj68-h5vbk"
Oct 30 00:43:54.045341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.044850    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wmj68-h5vbk" event={"ID":"15ba866d-3fba-4bf9-beec-14ecda492c3f","Type":"ContainerDied","Data":"dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7"}
Oct 30 00:43:54.045341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.044910    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wmj68-h5vbk" event={"ID":"15ba866d-3fba-4bf9-beec-14ecda492c3f","Type":"ContainerDied","Data":"6a7587e96d9aa2ca3dba26ed5500760dd5df7488d4cf688189d237032d408cb2"}
Oct 30 00:43:54.045341 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.044941    5104 scope.go:122] "RemoveContainer" containerID="dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7"
Oct 30 00:43:54.046368 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.046349    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:43:54.046977 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.046956    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wjf" event={"ID":"81da025c-1894-43c8-b83a-0b80409656ac","Type":"ContainerStarted","Data":"e59b87ad4bc7499143a69eb11dcc31b83bc56f899e7a74a9dfaa8d65e3f2e763"}
Oct 30 00:43:54.054450 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.054426    5104 scope.go:122] "RemoveContainer" containerID="dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7"
Oct 30 00:43:54.054758 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:43:54.054741    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7\": container with ID starting with dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7 not found: ID does not exist" containerID="dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7"
Oct 30 00:43:54.054830 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.054764    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7"} err="failed to get container status \"dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7\": rpc error: code = NotFound desc = could not find container \"dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7\": container with ID starting with dee03c6cbd76c7ac039b29f950d13ca61215b85c341f92964199bdc4748d3fc7 not found: ID does not exist"
Oct 30 00:43:54.111409 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.111376    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wmj68-h5vbk"]
Oct 30 00:43:54.114927 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.114892    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-wmj68-h5vbk"]
Oct 30 00:43:54.364768 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.364741    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wccd7-qxqqb"]
Oct 30 00:43:54.367393 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:43:54.367360    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b482a7_e9e0_4787_96f3_8949ab725a4d.slice/crio-03c6594f408773a8b84c35808842d9d11990717c5404f3dfae782846084b5d92 WatchSource:0}: Error finding container 03c6594f408773a8b84c35808842d9d11990717c5404f3dfae782846084b5d92: Status 404 returned error can't find the container with id 03c6594f408773a8b84c35808842d9d11990717c5404f3dfae782846084b5d92
Oct 30 00:43:54.574648 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:54.574612    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ba866d-3fba-4bf9-beec-14ecda492c3f" path="/var/lib/kubelet/pods/15ba866d-3fba-4bf9-beec-14ecda492c3f/volumes"
Oct 30 00:43:55.053630 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:55.053595    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wccd7-qxqqb" event={"ID":"68b482a7-e9e0-4787-96f3-8949ab725a4d","Type":"ContainerStarted","Data":"c4ad47f2faf09ec39089db2eb5b4049039e7a0918e4a86349a1120d8e5a7737c"}
Oct 30 00:43:55.054055 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:55.053646    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wccd7-qxqqb" event={"ID":"68b482a7-e9e0-4787-96f3-8949ab725a4d","Type":"ContainerStarted","Data":"03c6594f408773a8b84c35808842d9d11990717c5404f3dfae782846084b5d92"}
Oct 30 00:43:55.089505 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:43:55.089430    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wccd7-qxqqb" podStartSLOduration=2.089417717 podStartE2EDuration="2.089417717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:55.088630603 +0000 UTC m=+76.636321100" watchObservedRunningTime="2026-10-30 00:43:55.089417717 +0000 UTC m=+76.637108213"
Oct 30 00:44:05.095590 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:44:05.095551    5104 generic.go:356] "Generic (PLEG): container finished" podID="db543397-1d27-4098-a794-7a57aaa7904b" containerID="0d9d525349eeef431f3d58cce30db1c666207165ee362614e598fdb220b9eaa5" exitCode=0
Oct 30 00:44:05.096066 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:44:05.095624    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f" event={"ID":"db543397-1d27-4098-a794-7a57aaa7904b","Type":"ContainerDied","Data":"0d9d525349eeef431f3d58cce30db1c666207165ee362614e598fdb220b9eaa5"}
Oct 30 00:44:05.096066 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:44:05.095948    5104 scope.go:122] "RemoveContainer" containerID="0d9d525349eeef431f3d58cce30db1c666207165ee362614e598fdb220b9eaa5"
Oct 30 00:44:06.101657 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:44:06.101621    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-46g6f" event={"ID":"db543397-1d27-4098-a794-7a57aaa7904b","Type":"ContainerStarted","Data":"058c9e08291f557093fdbbdea663a0a904c47d173f116999a972cbae61cb1a59"}
Oct 30 00:44:09.102593 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:44:09.102554    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:44:39.104690 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:44:39.104645    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:45:09.102318 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:45:09.102230    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:45:18.397013 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:18.396919    5104 generic.go:356] "Generic (PLEG): container finished" podID="68b482a7-e9e0-4787-96f3-8949ab725a4d" containerID="c4ad47f2faf09ec39089db2eb5b4049039e7a0918e4a86349a1120d8e5a7737c" exitCode=0
Oct 30 00:45:18.397013 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:18.396993    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wccd7-qxqqb" event={"ID":"68b482a7-e9e0-4787-96f3-8949ab725a4d","Type":"ContainerDied","Data":"c4ad47f2faf09ec39089db2eb5b4049039e7a0918e4a86349a1120d8e5a7737c"}
Oct 30 00:45:19.526559 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.526531    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:45:19.530609 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.530588    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b482a7-e9e0-4787-96f3-8949ab725a4d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68b482a7-e9e0-4787-96f3-8949ab725a4d-archives-path\") pod \"68b482a7-e9e0-4787-96f3-8949ab725a4d\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") "
Oct 30 00:45:19.530728 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.530672    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-service-ca-bundle\") pod \"68b482a7-e9e0-4787-96f3-8949ab725a4d\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") "
Oct 30 00:45:19.530728 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.530700    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-trusted-ca-bundle\") pod \"68b482a7-e9e0-4787-96f3-8949ab725a4d\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") "
Oct 30 00:45:19.530928 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.530905    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b482a7-e9e0-4787-96f3-8949ab725a4d-archives-path" pod "68b482a7-e9e0-4787-96f3-8949ab725a4d" (UID: "68b482a7-e9e0-4787-96f3-8949ab725a4d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:45:19.531006 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.530986    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-service-ca-bundle" pod "68b482a7-e9e0-4787-96f3-8949ab725a4d" (UID: "68b482a7-e9e0-4787-96f3-8949ab725a4d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:19.531125 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.531106    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-trusted-ca-bundle" pod "68b482a7-e9e0-4787-96f3-8949ab725a4d" (UID: "68b482a7-e9e0-4787-96f3-8949ab725a4d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:19.631694 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.631657    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b482a7-e9e0-4787-96f3-8949ab725a4d-kube-api-access-prvjn\" (UniqueName: \"kubernetes.io/projected/68b482a7-e9e0-4787-96f3-8949ab725a4d-kube-api-access-prvjn\") pod \"68b482a7-e9e0-4787-96f3-8949ab725a4d\" (UID: \"68b482a7-e9e0-4787-96f3-8949ab725a4d\") "
Oct 30 00:45:19.631889 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.631802    5104 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-service-ca-bundle\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:45:19.631889 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.631814    5104 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68b482a7-e9e0-4787-96f3-8949ab725a4d-trusted-ca-bundle\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:45:19.631889 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.631824    5104 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/68b482a7-e9e0-4787-96f3-8949ab725a4d-archives-path\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:45:19.633817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.633794    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b482a7-e9e0-4787-96f3-8949ab725a4d-kube-api-access-prvjn" pod "68b482a7-e9e0-4787-96f3-8949ab725a4d" (UID: "68b482a7-e9e0-4787-96f3-8949ab725a4d"). InnerVolumeSpecName "kube-api-access-prvjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:19.733015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:19.732980    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvjn\" (UniqueName: \"kubernetes.io/projected/68b482a7-e9e0-4787-96f3-8949ab725a4d-kube-api-access-prvjn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:45:20.406390 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:20.406358    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wccd7-qxqqb"
Oct 30 00:45:20.406585 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:20.406396    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wccd7-qxqqb" event={"ID":"68b482a7-e9e0-4787-96f3-8949ab725a4d","Type":"ContainerDied","Data":"03c6594f408773a8b84c35808842d9d11990717c5404f3dfae782846084b5d92"}
Oct 30 00:45:20.406585 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:45:20.406442    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c6594f408773a8b84c35808842d9d11990717c5404f3dfae782846084b5d92"
Oct 30 00:45:39.102560 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:45:39.102523    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:46:09.102662 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:46:09.102618    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:46:39.102553 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:46:39.102511    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:47:09.102051 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:47:09.102010    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:47:38.595407 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:38.595375    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:47:38.598242 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:38.598215    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:47:38.603048 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:38.603024    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:47:38.605508 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:38.605492    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:47:39.102553 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:47:39.102519    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:47:49.773432 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.773350    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"]
Oct 30 00:47:49.773846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.773688    5104 cpu_manager.go:414] "containerMap: removing container" podUID="68b482a7-e9e0-4787-96f3-8949ab725a4d" containerName="insights-gathering"
Oct 30 00:47:49.773846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.773699    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b482a7-e9e0-4787-96f3-8949ab725a4d" containerName="insights-gathering"
Oct 30 00:47:49.773846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.773718    5104 cpu_manager.go:414] "containerMap: removing container" podUID="15ba866d-3fba-4bf9-beec-14ecda492c3f" containerName="insights-gathering"
Oct 30 00:47:49.773846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.773723    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ba866d-3fba-4bf9-beec-14ecda492c3f" containerName="insights-gathering"
Oct 30 00:47:49.773846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.773783    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ba866d-3fba-4bf9-beec-14ecda492c3f" containerName="insights-gathering"
Oct 30 00:47:49.773846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.773795    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b482a7-e9e0-4787-96f3-8949ab725a4d" containerName="insights-gathering"
Oct 30 00:47:49.776941 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.776908    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:49.793630 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.793604    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:47:49.804731 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.804683    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:47:49.804878 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.804743    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:47:49.804878 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.804805    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:47:49.804878 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.804871    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h7rmk\""
Oct 30 00:47:49.805041 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.804966    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:47:49.807007 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.806984    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"]
Oct 30 00:47:49.914914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.914878    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:49.915134 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.914939    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bb17fa7-7f07-436d-891d-b1af006715c5-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:49.915134 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.914963    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/6bb17fa7-7f07-436d-891d-b1af006715c5-kube-api-access-swt9j\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:49.915134 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.915073    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bb17fa7-7f07-436d-891d-b1af006715c5-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:49.915278 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.915188    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:49.915278 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:49.915223    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.016275 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.016239    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bb17fa7-7f07-436d-891d-b1af006715c5-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.016275 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.016283    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.016547 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.016398    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.016547 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.016486    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.016658 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.016584    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bb17fa7-7f07-436d-891d-b1af006715c5-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.016658 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.016640    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/6bb17fa7-7f07-436d-891d-b1af006715c5-kube-api-access-swt9j\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.016783 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.016763    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bb17fa7-7f07-436d-891d-b1af006715c5-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.018988 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.018954    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bb17fa7-7f07-436d-891d-b1af006715c5-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.019136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.019089    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.019136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.019101    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.019248 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.019171    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb17fa7-7f07-436d-891d-b1af006715c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.039348 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.039272    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/6bb17fa7-7f07-436d-891d-b1af006715c5-kube-api-access-swt9j\") pod \"opendatahub-operator-controller-manager-84dd548884-dqmtx\" (UID: \"6bb17fa7-7f07-436d-891d-b1af006715c5\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.097998 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.097960    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:50.433699 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.433673    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"]
Oct 30 00:47:50.437013 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:47:50.436980    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb17fa7_7f07_436d_891d_b1af006715c5.slice/crio-b7b74e2c5c61411186f339db0e8b326940eaddfed4844cb274a0bc9d6d06d787 WatchSource:0}: Error finding container b7b74e2c5c61411186f339db0e8b326940eaddfed4844cb274a0bc9d6d06d787: Status 404 returned error can't find the container with id b7b74e2c5c61411186f339db0e8b326940eaddfed4844cb274a0bc9d6d06d787
Oct 30 00:47:50.438607 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:50.438591    5104 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:47:51.017563 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:51.017523    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx" event={"ID":"6bb17fa7-7f07-436d-891d-b1af006715c5","Type":"ContainerStarted","Data":"b7b74e2c5c61411186f339db0e8b326940eaddfed4844cb274a0bc9d6d06d787"}
Oct 30 00:47:54.033525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:54.033426    5104 generic.go:356] "Generic (PLEG): container finished" podID="6bb17fa7-7f07-436d-891d-b1af006715c5" containerID="1b4c42bf8fac64f982a280f34792e7b98c5cc9a41654d86768303d0f74eb34fb" exitCode=0
Oct 30 00:47:54.033525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:54.033482    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx" event={"ID":"6bb17fa7-7f07-436d-891d-b1af006715c5","Type":"ContainerDied","Data":"1b4c42bf8fac64f982a280f34792e7b98c5cc9a41654d86768303d0f74eb34fb"}
Oct 30 00:47:55.040156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:55.040123    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx" event={"ID":"6bb17fa7-7f07-436d-891d-b1af006715c5","Type":"ContainerStarted","Data":"2083e6d8a3a0c4c23619b74df835c33817ddf64128b47d20d312bc66b43b17b2"}
Oct 30 00:47:55.040586 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:55.040210    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:47:55.113135 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:47:55.113080    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx" podStartSLOduration=2.480845998 podStartE2EDuration="6.113064673s" totalImagesPullingTime="3.165487431s" totalInitContainerRuntime="466.731244ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:55.101219839 +0000 UTC m=+316.648910335" watchObservedRunningTime="2026-10-30 00:47:55.113064673 +0000 UTC m=+316.660755163"
Oct 30 00:48:06.046382 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:06.046349    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-dqmtx"
Oct 30 00:48:09.102769 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:48:09.102731    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:48:10.783724 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.783691    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"]
Oct 30 00:48:10.795175 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.795147    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.805610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.805580    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"]
Oct 30 00:48:10.815932 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.815887    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 00:48:10.815932 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.815914    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7qwfp\""
Oct 30 00:48:10.815932 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.815914    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 00:48:10.870811 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.870774    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tmp\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.870811 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.870815    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.871023 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.870842    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjtt\" (UniqueName: \"kubernetes.io/projected/0fa71a9d-efce-4f19-addd-c6a2c38b5988-kube-api-access-7jjtt\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.971395 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.971355    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.971642 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.971410    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjtt\" (UniqueName: \"kubernetes.io/projected/0fa71a9d-efce-4f19-addd-c6a2c38b5988-kube-api-access-7jjtt\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.971642 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.971541    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tmp\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.971642 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:48:10.971555    5104 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Oct 30 00:48:10.971642 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:48:10.971634    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tls-certs podName:0fa71a9d-efce-4f19-addd-c6a2c38b5988 nodeName:}" failed. No retries permitted until 2026-10-30 00:48:11.471613288 +0000 UTC m=+333.019303785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tls-certs") pod "kube-auth-proxy-6cfd9d8567-cqfzb" (UID: "0fa71a9d-efce-4f19-addd-c6a2c38b5988") : secret "kube-auth-proxy-tls" not found
Oct 30 00:48:10.974022 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.973996    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tmp\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:10.990729 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:10.990699    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjtt\" (UniqueName: \"kubernetes.io/projected/0fa71a9d-efce-4f19-addd-c6a2c38b5988-kube-api-access-7jjtt\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:11.476738 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:11.476694    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:11.479538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:11.479414    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa71a9d-efce-4f19-addd-c6a2c38b5988-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-cqfzb\" (UID: \"0fa71a9d-efce-4f19-addd-c6a2c38b5988\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:11.708308 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:11.708270    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"
Oct 30 00:48:12.006083 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.006045    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb"]
Oct 30 00:48:12.009882 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:48:12.009853    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa71a9d_efce_4f19_addd_c6a2c38b5988.slice/crio-4e0b8556347e763d46142056cdcd3512c05a5404f39beb3ce8939dcb92f1bc90 WatchSource:0}: Error finding container 4e0b8556347e763d46142056cdcd3512c05a5404f39beb3ce8939dcb92f1bc90: Status 404 returned error can't find the container with id 4e0b8556347e763d46142056cdcd3512c05a5404f39beb3ce8939dcb92f1bc90
Oct 30 00:48:12.017282 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.017260    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"]
Oct 30 00:48:12.022107 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.022086    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.039530 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.039505    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:48:12.045930 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.045905    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"]
Oct 30 00:48:12.084894 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.084858    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.084894 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.084907    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.085136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.084935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.085136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.085040    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clb54\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-kube-api-access-clb54\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.085136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.085074    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.085136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.085101    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.085338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.085143    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.090558 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.090531    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 00:48:12.090558 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.090537    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-txv2f\""
Oct 30 00:48:12.090738 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.090560    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 00:48:12.090738 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.090588    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 00:48:12.109042 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.109007    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb" event={"ID":"0fa71a9d-efce-4f19-addd-c6a2c38b5988","Type":"ContainerStarted","Data":"4e0b8556347e763d46142056cdcd3512c05a5404f39beb3ce8939dcb92f1bc90"}
Oct 30 00:48:12.186309 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.186271    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.186309 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.186313    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.186620 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.186337    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.186620 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.186401    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.186620 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.186441    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.186620 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.186501    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.186620 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.186543    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clb54\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-kube-api-access-clb54\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.187198 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.187163    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.188933 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.188912    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.189034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.188960    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.189142 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.189125    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.189292 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.189270    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.205077 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.205047    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.205381 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.205360    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clb54\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-kube-api-access-clb54\") pod \"istiod-openshift-gateway-7876bfcb74-zhsns\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.333912 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.333827    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:12.641415 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:12.641323    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"]
Oct 30 00:48:12.646604 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:48:12.646567    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2f7879_8101_41e3_b02c_fdfb50da1a3c.slice/crio-89f56b1186d2804ce863a430ebd346bb1268b550b3caa76faa69b41367fa2d62 WatchSource:0}: Error finding container 89f56b1186d2804ce863a430ebd346bb1268b550b3caa76faa69b41367fa2d62: Status 404 returned error can't find the container with id 89f56b1186d2804ce863a430ebd346bb1268b550b3caa76faa69b41367fa2d62
Oct 30 00:48:13.117124 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:13.117087    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns" event={"ID":"9d2f7879-8101-41e3-b02c-fdfb50da1a3c","Type":"ContainerStarted","Data":"89f56b1186d2804ce863a430ebd346bb1268b550b3caa76faa69b41367fa2d62"}
Oct 30 00:48:15.127550 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:15.127503    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb" event={"ID":"0fa71a9d-efce-4f19-addd-c6a2c38b5988","Type":"ContainerStarted","Data":"6d70084f07ae7070d3e5855756f7e4085e75a0e932e51ad902c5e7a3369ad91f"}
Oct 30 00:48:15.163835 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:15.163776    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-cqfzb" podStartSLOduration=2.529186911 podStartE2EDuration="5.163761851s" totalImagesPullingTime="2.63457494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:15.162619806 +0000 UTC m=+336.710310307" watchObservedRunningTime="2026-10-30 00:48:15.163761851 +0000 UTC m=+336.711452363"
Oct 30 00:48:15.187789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:15.187756    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:48:15.187894 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:15.187822    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:48:16.134203 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:16.134164    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns" event={"ID":"9d2f7879-8101-41e3-b02c-fdfb50da1a3c","Type":"ContainerStarted","Data":"0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f"}
Oct 30 00:48:16.134694 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:16.134501    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:16.136651 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:16.136618    5104 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-zhsns container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:48:16.136800 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:16.136670    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns" podUID="9d2f7879-8101-41e3-b02c-fdfb50da1a3c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:48:16.201914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:16.201849    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns" podStartSLOduration=2.664784442 podStartE2EDuration="5.201827532s" totalImagesPullingTime="2.53704309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:16.189424951 +0000 UTC m=+337.737115446" watchObservedRunningTime="2026-10-30 00:48:16.201827532 +0000 UTC m=+337.749518053"
Oct 30 00:48:17.139654 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:17.139619    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:48:27.778719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.778654    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"]
Oct 30 00:48:27.782444 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.782419    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:27.796108 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.796087    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-6bbwf\""
Oct 30 00:48:27.831323 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.831285    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 00:48:27.832889 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.832865    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"]
Oct 30 00:48:27.928576 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.928535    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlrr\" (UniqueName: \"kubernetes.io/projected/6df220f5-b138-4085-b1b3-6f9f6f114abd-kube-api-access-5hlrr\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:27.928780 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.928625    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df220f5-b138-4085-b1b3-6f9f6f114abd-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:27.928780 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.928657    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df220f5-b138-4085-b1b3-6f9f6f114abd-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:27.928780 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:27.928683    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6df220f5-b138-4085-b1b3-6f9f6f114abd-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.030213 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.030106    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlrr\" (UniqueName: \"kubernetes.io/projected/6df220f5-b138-4085-b1b3-6f9f6f114abd-kube-api-access-5hlrr\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.030213 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.030203    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df220f5-b138-4085-b1b3-6f9f6f114abd-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.030438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.030237    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df220f5-b138-4085-b1b3-6f9f6f114abd-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.030438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.030264    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6df220f5-b138-4085-b1b3-6f9f6f114abd-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.030727 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.030702    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df220f5-b138-4085-b1b3-6f9f6f114abd-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.030798 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.030750    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6df220f5-b138-4085-b1b3-6f9f6f114abd-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.031115 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.031096    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df220f5-b138-4085-b1b3-6f9f6f114abd-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.048131 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.048103    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlrr\" (UniqueName: \"kubernetes.io/projected/6df220f5-b138-4085-b1b3-6f9f6f114abd-kube-api-access-5hlrr\") pod \"kserve-module-controller-manager-5cfb77bf8c-z98l9\" (UID: \"6df220f5-b138-4085-b1b3-6f9f6f114abd\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.095815 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.095773    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:28.397853 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:28.397767    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"]
Oct 30 00:48:28.402041 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:48:28.402013    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df220f5_b138_4085_b1b3_6f9f6f114abd.slice/crio-e8bbb9fb424b5c5401eced361fdd534273774986c93cf2985c54a46ab9bfcf42 WatchSource:0}: Error finding container e8bbb9fb424b5c5401eced361fdd534273774986c93cf2985c54a46ab9bfcf42: Status 404 returned error can't find the container with id e8bbb9fb424b5c5401eced361fdd534273774986c93cf2985c54a46ab9bfcf42
Oct 30 00:48:29.192203 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:29.192168    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9" event={"ID":"6df220f5-b138-4085-b1b3-6f9f6f114abd","Type":"ContainerStarted","Data":"e8bbb9fb424b5c5401eced361fdd534273774986c93cf2985c54a46ab9bfcf42"}
Oct 30 00:48:30.197835 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:30.197796    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9" event={"ID":"6df220f5-b138-4085-b1b3-6f9f6f114abd","Type":"ContainerStarted","Data":"a42cb987603ef95fa9ba357415b63d1351e865ba03651e4e2862a25c6ed9ec71"}
Oct 30 00:48:30.198268 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:30.197935    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:48:30.246621 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:30.246505    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9" podStartSLOduration=1.656736167 podStartE2EDuration="3.24648823s" totalImagesPullingTime="1.589752063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:30.246360317 +0000 UTC m=+351.794050813" watchObservedRunningTime="2026-10-30 00:48:30.24648823 +0000 UTC m=+351.794178726"
Oct 30 00:48:39.103040 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:48:39.102999    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:48:51.203375 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:48:51.203340    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-z98l9"
Oct 30 00:49:09.102717 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:49:09.102675    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:49:13.150841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.150808    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"]
Oct 30 00:49:13.158233 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.158206    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.186015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.185981    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"]
Oct 30 00:49:13.220851 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.220804    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mdx\" (UniqueName: \"kubernetes.io/projected/56d942cc-a33a-4bf9-b0f8-baa86d240b19-kube-api-access-b4mdx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.220851 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.220855    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.221064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.220968    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.221064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.221009    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d942cc-a33a-4bf9-b0f8-baa86d240b19-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.221151 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.221111    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.221216 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.221200    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.221267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.221253    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.322536 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.322482    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.322536 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.322538    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d942cc-a33a-4bf9-b0f8-baa86d240b19-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.322817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.322611    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.322817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.322662    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.322817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.322714    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.322817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.322743    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mdx\" (UniqueName: \"kubernetes.io/projected/56d942cc-a33a-4bf9-b0f8-baa86d240b19-kube-api-access-b4mdx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.322817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.322773    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.323475 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.323433    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.325223 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.325191    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.325354 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.325235    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.325354 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.325327    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d942cc-a33a-4bf9-b0f8-baa86d240b19-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.325490 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.325448    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.353420 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.353387    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d942cc-a33a-4bf9-b0f8-baa86d240b19-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.353420 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.353418    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mdx\" (UniqueName: \"kubernetes.io/projected/56d942cc-a33a-4bf9-b0f8-baa86d240b19-kube-api-access-b4mdx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bdqg\" (UID: \"56d942cc-a33a-4bf9-b0f8-baa86d240b19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.470275 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.470240    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:13.772164 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.772080    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"]
Oct 30 00:49:13.775777 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:49:13.775748    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d942cc_a33a_4bf9_b0f8_baa86d240b19.slice/crio-0aa222858e899a17186c57085c527da5b4e8bb58152c741022f1120a41ed1069 WatchSource:0}: Error finding container 0aa222858e899a17186c57085c527da5b4e8bb58152c741022f1120a41ed1069: Status 404 returned error can't find the container with id 0aa222858e899a17186c57085c527da5b4e8bb58152c741022f1120a41ed1069
Oct 30 00:49:13.777903 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.777858    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:13.777996 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:13.777950    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:49:14.395132 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:14.395092    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg" event={"ID":"56d942cc-a33a-4bf9-b0f8-baa86d240b19","Type":"ContainerStarted","Data":"a5b13f9ef00787fba2b13bbce0a367120178c558b35ce92449ce981b7d77bca7"}
Oct 30 00:49:14.395649 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:14.395187    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg" event={"ID":"56d942cc-a33a-4bf9-b0f8-baa86d240b19","Type":"ContainerStarted","Data":"0aa222858e899a17186c57085c527da5b4e8bb58152c741022f1120a41ed1069"}
Oct 30 00:49:14.395741 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:14.395718    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:14.397169 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:14.397137    5104 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6bdqg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:49:14.397284 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:14.397206    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg" podUID="56d942cc-a33a-4bf9-b0f8-baa86d240b19" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:49:14.437703 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:14.437645    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg" podStartSLOduration=1.4376331150000001 podStartE2EDuration="1.437633115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:14.437022668 +0000 UTC m=+395.984713163" watchObservedRunningTime="2026-10-30 00:49:14.437633115 +0000 UTC m=+395.985323608"
Oct 30 00:49:15.401143 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.401116    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bdqg"
Oct 30 00:49:15.471662 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.471623    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"]
Oct 30 00:49:15.471994 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.471933    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns" podUID="9d2f7879-8101-41e3-b02c-fdfb50da1a3c" containerName="discovery" containerID="cri-o://0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f" gracePeriod=30
Oct 30 00:49:15.718452 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.718424    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:49:15.744304 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744274    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-cacerts\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-cacerts\") pod \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") "
Oct 30 00:49:15.744431 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744312    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-token\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-token\") pod \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") "
Oct 30 00:49:15.744431 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744336    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-kube-api-access-clb54\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-kube-api-access-clb54\") pod \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") "
Oct 30 00:49:15.744431 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744368    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-local-certs\") pod \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") "
Oct 30 00:49:15.744431 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744400    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-ca-configmap\") pod \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") "
Oct 30 00:49:15.744431 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744425    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-kubeconfig\") pod \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") "
Oct 30 00:49:15.744724 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744501    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-dns-cert\") pod \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\" (UID: \"9d2f7879-8101-41e3-b02c-fdfb50da1a3c\") "
Oct 30 00:49:15.744908 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.744874    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-ca-configmap" pod "9d2f7879-8101-41e3-b02c-fdfb50da1a3c" (UID: "9d2f7879-8101-41e3-b02c-fdfb50da1a3c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:15.747108 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.747075    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-token" pod "9d2f7879-8101-41e3-b02c-fdfb50da1a3c" (UID: "9d2f7879-8101-41e3-b02c-fdfb50da1a3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:15.747226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.747102    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-cacerts" pod "9d2f7879-8101-41e3-b02c-fdfb50da1a3c" (UID: "9d2f7879-8101-41e3-b02c-fdfb50da1a3c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:15.747226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.747120    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-local-certs" pod "9d2f7879-8101-41e3-b02c-fdfb50da1a3c" (UID: "9d2f7879-8101-41e3-b02c-fdfb50da1a3c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:49:15.747226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.747127    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-dns-cert" pod "9d2f7879-8101-41e3-b02c-fdfb50da1a3c" (UID: "9d2f7879-8101-41e3-b02c-fdfb50da1a3c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:15.747331 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.747230    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-kube-api-access-clb54" pod "9d2f7879-8101-41e3-b02c-fdfb50da1a3c" (UID: "9d2f7879-8101-41e3-b02c-fdfb50da1a3c"). InnerVolumeSpecName "kube-api-access-clb54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:15.747331 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.747264    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-kubeconfig" pod "9d2f7879-8101-41e3-b02c-fdfb50da1a3c" (UID: "9d2f7879-8101-41e3-b02c-fdfb50da1a3c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:15.846149 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.846107    5104 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-cacerts\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:49:15.846149 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.846141    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:49:15.846149 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.846153    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clb54\" (UniqueName: \"kubernetes.io/projected/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-kube-api-access-clb54\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:49:15.846382 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.846163    5104 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-local-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:49:15.846382 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.846172    5104 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-ca-configmap\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:49:15.846382 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.846181    5104 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-kubeconfig\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:49:15.846382 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:15.846190    5104 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d2f7879-8101-41e3-b02c-fdfb50da1a3c-istio-csr-dns-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:49:16.406430 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.406396    5104 generic.go:356] "Generic (PLEG): container finished" podID="9d2f7879-8101-41e3-b02c-fdfb50da1a3c" containerID="0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f" exitCode=0
Oct 30 00:49:16.406855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.406478    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"
Oct 30 00:49:16.406855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.406489    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns" event={"ID":"9d2f7879-8101-41e3-b02c-fdfb50da1a3c","Type":"ContainerDied","Data":"0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f"}
Oct 30 00:49:16.406855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.406551    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns" event={"ID":"9d2f7879-8101-41e3-b02c-fdfb50da1a3c","Type":"ContainerDied","Data":"89f56b1186d2804ce863a430ebd346bb1268b550b3caa76faa69b41367fa2d62"}
Oct 30 00:49:16.406855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.406581    5104 scope.go:122] "RemoveContainer" containerID="0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f"
Oct 30 00:49:16.415416 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.415395    5104 scope.go:122] "RemoveContainer" containerID="0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f"
Oct 30 00:49:16.415713 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:49:16.415694    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f\": container with ID starting with 0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f not found: ID does not exist" containerID="0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f"
Oct 30 00:49:16.415804 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.415721    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f"} err="failed to get container status \"0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f\": rpc error: code = NotFound desc = could not find container \"0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f\": container with ID starting with 0fc2251e7a36de703610ef6da190b1fd399ae02f743a36447442e051bfa7ee9f not found: ID does not exist"
Oct 30 00:49:16.465482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.465407    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"]
Oct 30 00:49:16.476487 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.476443    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zhsns"]
Oct 30 00:49:16.575161 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:49:16.575132    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2f7879-8101-41e3-b02c-fdfb50da1a3c" path="/var/lib/kubelet/pods/9d2f7879-8101-41e3-b02c-fdfb50da1a3c/volumes"
Oct 30 00:49:39.102833 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:49:39.102793    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:50:09.102179 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:50:09.102140    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:50:23.419147 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.419110    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"]
Oct 30 00:50:23.419678 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.419502    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9d2f7879-8101-41e3-b02c-fdfb50da1a3c" containerName="discovery"
Oct 30 00:50:23.419678 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.419514    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2f7879-8101-41e3-b02c-fdfb50da1a3c" containerName="discovery"
Oct 30 00:50:23.419678 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.419569    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2f7879-8101-41e3-b02c-fdfb50da1a3c" containerName="discovery"
Oct 30 00:50:23.422652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.422632    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:23.449280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.449249    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 00:50:23.449280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.449265    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 00:50:23.449513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.449297    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v6n6c\""
Oct 30 00:50:23.450807 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.450790    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"]
Oct 30 00:50:23.546567 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.546531    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43a28c9a-1ef4-4899-ba27-bb22f442bac1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:23.546730 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.546688    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crqln\" (UniqueName: \"kubernetes.io/projected/43a28c9a-1ef4-4899-ba27-bb22f442bac1-kube-api-access-crqln\") pod \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:23.647159 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.647117    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crqln\" (UniqueName: \"kubernetes.io/projected/43a28c9a-1ef4-4899-ba27-bb22f442bac1-kube-api-access-crqln\") pod \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:23.647318 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.647167    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43a28c9a-1ef4-4899-ba27-bb22f442bac1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:23.647639 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.647610    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43a28c9a-1ef4-4899-ba27-bb22f442bac1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:23.666038 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.666012    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crqln\" (UniqueName: \"kubernetes.io/projected/43a28c9a-1ef4-4899-ba27-bb22f442bac1-kube-api-access-crqln\") pod \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:23.735931 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:23.735854    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:24.050433 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:24.050289    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"]
Oct 30 00:50:24.054095 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:50:24.054068    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a28c9a_1ef4_4899_ba27_bb22f442bac1.slice/crio-e05320e88a85afb7f99183bbcfabf8310d723cfb5e6659e7ef3d607a9aef07d4 WatchSource:0}: Error finding container e05320e88a85afb7f99183bbcfabf8310d723cfb5e6659e7ef3d607a9aef07d4: Status 404 returned error can't find the container with id e05320e88a85afb7f99183bbcfabf8310d723cfb5e6659e7ef3d607a9aef07d4
Oct 30 00:50:24.704581 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:24.704547    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b" event={"ID":"43a28c9a-1ef4-4899-ba27-bb22f442bac1","Type":"ContainerStarted","Data":"e05320e88a85afb7f99183bbcfabf8310d723cfb5e6659e7ef3d607a9aef07d4"}
Oct 30 00:50:26.060420 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.060383    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"]
Oct 30 00:50:26.065122 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.065099    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"
Oct 30 00:50:26.093983 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.093952    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cvc46\""
Oct 30 00:50:26.095673 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.095647    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"]
Oct 30 00:50:26.169482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.169419    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6j9\" (UniqueName: \"kubernetes.io/projected/8afd8118-e084-42a5-baf0-dddd04b92b7f-kube-api-access-fv6j9\") pod \"authorino-operator-68d6dd76dc-6r7jm\" (UID: \"8afd8118-e084-42a5-baf0-dddd04b92b7f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"
Oct 30 00:50:26.270887 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.270849    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6j9\" (UniqueName: \"kubernetes.io/projected/8afd8118-e084-42a5-baf0-dddd04b92b7f-kube-api-access-fv6j9\") pod \"authorino-operator-68d6dd76dc-6r7jm\" (UID: \"8afd8118-e084-42a5-baf0-dddd04b92b7f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"
Oct 30 00:50:26.290850 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.290817    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6j9\" (UniqueName: \"kubernetes.io/projected/8afd8118-e084-42a5-baf0-dddd04b92b7f-kube-api-access-fv6j9\") pod \"authorino-operator-68d6dd76dc-6r7jm\" (UID: \"8afd8118-e084-42a5-baf0-dddd04b92b7f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"
Oct 30 00:50:26.378142 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.378049    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"
Oct 30 00:50:26.702869 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.702839    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"]
Oct 30 00:50:26.716924 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:26.716892    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm" event={"ID":"8afd8118-e084-42a5-baf0-dddd04b92b7f","Type":"ContainerStarted","Data":"3b7d50dbdac95f665be863ca9b592ecba321b23d1a2761c28cb5cae5c4dc95b4"}
Oct 30 00:50:29.733824 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:29.733689    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm" event={"ID":"8afd8118-e084-42a5-baf0-dddd04b92b7f","Type":"ContainerStarted","Data":"598272bdcc3894f5c3cb3b721512c10bbc42ac78e4d2b90407e683e4226898f6"}
Oct 30 00:50:29.733824 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:29.733782    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"
Oct 30 00:50:29.773854 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:29.773797    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm" podStartSLOduration=1.057339041 podStartE2EDuration="3.77378163s" totalImagesPullingTime="2.716442589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:29.773732687 +0000 UTC m=+471.321423185" watchObservedRunningTime="2026-10-30 00:50:29.77378163 +0000 UTC m=+471.321472123"
Oct 30 00:50:39.102870 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:50:39.102835    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:50:40.740951 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:40.740919    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-6r7jm"
Oct 30 00:50:48.821600 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:48.821485    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b" event={"ID":"43a28c9a-1ef4-4899-ba27-bb22f442bac1","Type":"ContainerStarted","Data":"422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6"}
Oct 30 00:50:48.821600 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:48.821567    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:50:48.883727 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:48.883676    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b" podStartSLOduration=1.3734376240000001 podStartE2EDuration="25.883662082s" totalImagesPullingTime="24.510224458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:48.882657363 +0000 UTC m=+490.430347859" watchObservedRunningTime="2026-10-30 00:50:48.883662082 +0000 UTC m=+490.431352577"
Oct 30 00:50:59.828718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:50:59.828676    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:03.318081 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.318046    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"]
Oct 30 00:51:03.318555 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.318263    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" containerName="manager" containerID="cri-o://422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6" gracePeriod=2
Oct 30 00:51:03.319650 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.319574    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"]
Oct 30 00:51:03.331493 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:03.331433    5104 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" is forbidden: User \"system:node:ip-10-0-130-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-169.ec2.internal' and this object" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:03.332483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.332446    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"]
Oct 30 00:51:03.333107 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.333080    5104 cpu_manager.go:414] "containerMap: removing container" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" containerName="manager"
Oct 30 00:51:03.333107 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.333110    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" containerName="manager"
Oct 30 00:51:03.333290 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.333235    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" containerName="manager"
Oct 30 00:51:03.338659 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.338639    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.354276 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:03.354237    5104 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" is forbidden: User \"system:node:ip-10-0-130-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-169.ec2.internal' and this object" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:03.384710 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.384673    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"]
Oct 30 00:51:03.401930 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.401895    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5sgv\" (UniqueName: \"kubernetes.io/projected/c35de467-f409-4879-ae8e-a825d572b373-kube-api-access-c5sgv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-l45n8\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.402119 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.402001    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c35de467-f409-4879-ae8e-a825d572b373-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-l45n8\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.503114 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.503075    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c35de467-f409-4879-ae8e-a825d572b373-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-l45n8\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.503277 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.503133    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5sgv\" (UniqueName: \"kubernetes.io/projected/c35de467-f409-4879-ae8e-a825d572b373-kube-api-access-c5sgv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-l45n8\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.503610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.503577    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c35de467-f409-4879-ae8e-a825d572b373-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-l45n8\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.523958 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.523925    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5sgv\" (UniqueName: \"kubernetes.io/projected/c35de467-f409-4879-ae8e-a825d572b373-kube-api-access-c5sgv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-l45n8\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.577240 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.577178    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:03.589815 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:03.589779    5104 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" is forbidden: User \"system:node:ip-10-0-130-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-169.ec2.internal' and this object" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:03.704713 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.704676    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a28c9a-1ef4-4899-ba27-bb22f442bac1-kube-api-access-crqln\" (UniqueName: \"kubernetes.io/projected/43a28c9a-1ef4-4899-ba27-bb22f442bac1-kube-api-access-crqln\") pod \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") "
Oct 30 00:51:03.704713 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.704724    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a28c9a-1ef4-4899-ba27-bb22f442bac1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43a28c9a-1ef4-4899-ba27-bb22f442bac1-extensions-socket-volume\") pod \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\" (UID: \"43a28c9a-1ef4-4899-ba27-bb22f442bac1\") "
Oct 30 00:51:03.705241 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.705214    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a28c9a-1ef4-4899-ba27-bb22f442bac1-extensions-socket-volume" pod "43a28c9a-1ef4-4899-ba27-bb22f442bac1" (UID: "43a28c9a-1ef4-4899-ba27-bb22f442bac1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:51:03.706995 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.706966    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a28c9a-1ef4-4899-ba27-bb22f442bac1-kube-api-access-crqln" pod "43a28c9a-1ef4-4899-ba27-bb22f442bac1" (UID: "43a28c9a-1ef4-4899-ba27-bb22f442bac1"). InnerVolumeSpecName "kube-api-access-crqln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:03.717161 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.717135    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:03.805771 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.805739    5104 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43a28c9a-1ef4-4899-ba27-bb22f442bac1-extensions-socket-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:51:03.805771 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.805767    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crqln\" (UniqueName: \"kubernetes.io/projected/43a28c9a-1ef4-4899-ba27-bb22f442bac1-kube-api-access-crqln\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:51:03.888015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.887917    5104 generic.go:356] "Generic (PLEG): container finished" podID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" containerID="422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6" exitCode=0
Oct 30 00:51:03.888015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.887959    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:03.888188 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.888023    5104 scope.go:122] "RemoveContainer" containerID="422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6"
Oct 30 00:51:03.897729 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.897708    5104 scope.go:122] "RemoveContainer" containerID="422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6"
Oct 30 00:51:03.898015 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:03.897997    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6\": container with ID starting with 422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6 not found: ID does not exist" containerID="422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6"
Oct 30 00:51:03.898059 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:03.898027    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6"} err="failed to get container status \"422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6\": rpc error: code = NotFound desc = could not find container \"422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6\": container with ID starting with 422e1ece520f62152f987c0854e95375fe6245b2acd47dc0cd4ca7f03e5a8aa6 not found: ID does not exist"
Oct 30 00:51:03.900727 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:03.900705    5104 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" is forbidden: User \"system:node:ip-10-0-130-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-169.ec2.internal' and this object" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:03.913041 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:03.913015    5104 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-tcl8b\" is forbidden: User \"system:node:ip-10-0-130-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-169.ec2.internal' and this object" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-tcl8b"
Oct 30 00:51:04.034614 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:04.034576    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"]
Oct 30 00:51:04.038547 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:51:04.038518    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35de467_f409_4879_ae8e_a825d572b373.slice/crio-9c60b51e8e0180104bef31228ed7cdcd8511af018cfff21d7638c9dedd04e911 WatchSource:0}: Error finding container 9c60b51e8e0180104bef31228ed7cdcd8511af018cfff21d7638c9dedd04e911: Status 404 returned error can't find the container with id 9c60b51e8e0180104bef31228ed7cdcd8511af018cfff21d7638c9dedd04e911
Oct 30 00:51:04.575756 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:04.575719    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a28c9a-1ef4-4899-ba27-bb22f442bac1" path="/var/lib/kubelet/pods/43a28c9a-1ef4-4899-ba27-bb22f442bac1/volumes"
Oct 30 00:51:04.896347 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:04.896252    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8" event={"ID":"c35de467-f409-4879-ae8e-a825d572b373","Type":"ContainerStarted","Data":"4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a"}
Oct 30 00:51:04.896347 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:04.896309    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8" event={"ID":"c35de467-f409-4879-ae8e-a825d572b373","Type":"ContainerStarted","Data":"9c60b51e8e0180104bef31228ed7cdcd8511af018cfff21d7638c9dedd04e911"}
Oct 30 00:51:04.896600 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:04.896476    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:04.959093 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:04.959035    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8" podStartSLOduration=1.959020932 podStartE2EDuration="1.959020932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:04.946354436 +0000 UTC m=+506.494044931" watchObservedRunningTime="2026-10-30 00:51:04.959020932 +0000 UTC m=+506.506711428"
Oct 30 00:51:09.101848 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:09.101807    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:51:15.904144 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:15.904109    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:19.595525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.595489    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"]
Oct 30 00:51:19.596018 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.595829    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8" podUID="c35de467-f409-4879-ae8e-a825d572b373" containerName="manager" containerID="cri-o://4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a" gracePeriod=10
Oct 30 00:51:19.807556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.807512    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"]
Oct 30 00:51:19.811518 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.811495    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:19.834611 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.834572    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"]
Oct 30 00:51:19.849666 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.849598    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:19.949645 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.949613    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35de467-f409-4879-ae8e-a825d572b373-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c35de467-f409-4879-ae8e-a825d572b373-extensions-socket-volume\") pod \"c35de467-f409-4879-ae8e-a825d572b373\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") "
Oct 30 00:51:19.949645 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.949663    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35de467-f409-4879-ae8e-a825d572b373-kube-api-access-c5sgv\" (UniqueName: \"kubernetes.io/projected/c35de467-f409-4879-ae8e-a825d572b373-kube-api-access-c5sgv\") pod \"c35de467-f409-4879-ae8e-a825d572b373\" (UID: \"c35de467-f409-4879-ae8e-a825d572b373\") "
Oct 30 00:51:19.949873 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.949796    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdbz\" (UniqueName: \"kubernetes.io/projected/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-kube-api-access-dxdbz\") pod \"kuadrant-operator-controller-manager-57bb9bfff-pc7pj\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:19.949873 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.949829    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-pc7pj\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:19.950131 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.950103    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35de467-f409-4879-ae8e-a825d572b373-extensions-socket-volume" pod "c35de467-f409-4879-ae8e-a825d572b373" (UID: "c35de467-f409-4879-ae8e-a825d572b373"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:51:19.951947 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.951918    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35de467-f409-4879-ae8e-a825d572b373-kube-api-access-c5sgv" pod "c35de467-f409-4879-ae8e-a825d572b373" (UID: "c35de467-f409-4879-ae8e-a825d572b373"). InnerVolumeSpecName "kube-api-access-c5sgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:19.965742 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.965716    5104 generic.go:356] "Generic (PLEG): container finished" podID="c35de467-f409-4879-ae8e-a825d572b373" containerID="4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a" exitCode=0
Oct 30 00:51:19.965875 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.965755    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8" event={"ID":"c35de467-f409-4879-ae8e-a825d572b373","Type":"ContainerDied","Data":"4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a"}
Oct 30 00:51:19.965875 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.965759    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"
Oct 30 00:51:19.965875 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.965799    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8" event={"ID":"c35de467-f409-4879-ae8e-a825d572b373","Type":"ContainerDied","Data":"9c60b51e8e0180104bef31228ed7cdcd8511af018cfff21d7638c9dedd04e911"}
Oct 30 00:51:19.965875 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.965820    5104 scope.go:122] "RemoveContainer" containerID="4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a"
Oct 30 00:51:19.974491 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.974467    5104 scope.go:122] "RemoveContainer" containerID="4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a"
Oct 30 00:51:19.974749 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:19.974732    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a\": container with ID starting with 4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a not found: ID does not exist" containerID="4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a"
Oct 30 00:51:19.974803 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:19.974757    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a"} err="failed to get container status \"4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a\": rpc error: code = NotFound desc = could not find container \"4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a\": container with ID starting with 4c84482709d7be98c12ada9385f9fcc6a32dd524de4ed7ccc5a5354cfb9eb64a not found: ID does not exist"
Oct 30 00:51:20.014605 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.014568    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"]
Oct 30 00:51:20.018107 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.018081    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-l45n8"]
Oct 30 00:51:20.050785 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.050750    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdbz\" (UniqueName: \"kubernetes.io/projected/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-kube-api-access-dxdbz\") pod \"kuadrant-operator-controller-manager-57bb9bfff-pc7pj\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:20.050988 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.050795    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-pc7pj\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:20.050988 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.050835    5104 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c35de467-f409-4879-ae8e-a825d572b373-extensions-socket-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:51:20.050988 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.050849    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5sgv\" (UniqueName: \"kubernetes.io/projected/c35de467-f409-4879-ae8e-a825d572b373-kube-api-access-c5sgv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:51:20.051182 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.051164    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-pc7pj\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:20.072550 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.072521    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdbz\" (UniqueName: \"kubernetes.io/projected/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-kube-api-access-dxdbz\") pod \"kuadrant-operator-controller-manager-57bb9bfff-pc7pj\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:20.124774 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.124697    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:20.441757 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.441727    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"]
Oct 30 00:51:20.445486 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:51:20.445434    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b8f43d_70bd_42d4_b1b0_283fa053f8f4.slice/crio-47ba89f388f7a291b739f86ffefac5f68fc93fad89887fc0cdb74db3fa200ee0 WatchSource:0}: Error finding container 47ba89f388f7a291b739f86ffefac5f68fc93fad89887fc0cdb74db3fa200ee0: Status 404 returned error can't find the container with id 47ba89f388f7a291b739f86ffefac5f68fc93fad89887fc0cdb74db3fa200ee0
Oct 30 00:51:20.574975 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.574940    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35de467-f409-4879-ae8e-a825d572b373" path="/var/lib/kubelet/pods/c35de467-f409-4879-ae8e-a825d572b373/volumes"
Oct 30 00:51:20.972868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.972835    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj" event={"ID":"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4","Type":"ContainerStarted","Data":"a089eaa8decf5de1c03ff0c970e8477d6bc60111104a95bd472d702f4c44af14"}
Oct 30 00:51:20.973324 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.972886    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj" event={"ID":"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4","Type":"ContainerStarted","Data":"47ba89f388f7a291b739f86ffefac5f68fc93fad89887fc0cdb74db3fa200ee0"}
Oct 30 00:51:20.973324 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:20.972921    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:21.041185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:21.041121    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj" podStartSLOduration=2.041107181 podStartE2EDuration="2.041107181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:21.028650041 +0000 UTC m=+522.576340537" watchObservedRunningTime="2026-10-30 00:51:21.041107181 +0000 UTC m=+522.588797676"
Oct 30 00:51:31.981602 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:31.981572    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 00:51:39.102434 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:51:39.102387    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:51:52.433000 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.432963    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:51:52.433753 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.433730    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c35de467-f409-4879-ae8e-a825d572b373" containerName="manager"
Oct 30 00:51:52.433753 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.433750    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35de467-f409-4879-ae8e-a825d572b373" containerName="manager"
Oct 30 00:51:52.433925 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.433825    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35de467-f409-4879-ae8e-a825d572b373" containerName="manager"
Oct 30 00:51:52.436914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.436896    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:52.456515 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.456484    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 00:51:52.465122 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.465090    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gs4jx\""
Oct 30 00:51:52.466525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.466498    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:51:52.511238 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.511201    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:51:52.518155 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.518125    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrnpr\" (UniqueName: \"kubernetes.io/projected/fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e-kube-api-access-vrnpr\") pod \"limitador-limitador-7c9b87d574-76ntd\" (UID: \"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:52.518303 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.518166    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e-config-file\") pod \"limitador-limitador-7c9b87d574-76ntd\" (UID: \"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:52.618880 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.618843    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrnpr\" (UniqueName: \"kubernetes.io/projected/fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e-kube-api-access-vrnpr\") pod \"limitador-limitador-7c9b87d574-76ntd\" (UID: \"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:52.619052 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.618896    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e-config-file\") pod \"limitador-limitador-7c9b87d574-76ntd\" (UID: \"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:52.619564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.619546    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e-config-file\") pod \"limitador-limitador-7c9b87d574-76ntd\" (UID: \"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:52.636997 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.636965    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrnpr\" (UniqueName: \"kubernetes.io/projected/fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e-kube-api-access-vrnpr\") pod \"limitador-limitador-7c9b87d574-76ntd\" (UID: \"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:52.750064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:52.749980    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:53.039582 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:53.039504    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:51:53.043259 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:51:53.043225    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee88f31_2ff9_4d35_ae21_9fce7cbcdd8e.slice/crio-076d4a3353e38ffd0f0632a16ba84af39e216a4f2c73a466387bbad3bdbe9040 WatchSource:0}: Error finding container 076d4a3353e38ffd0f0632a16ba84af39e216a4f2c73a466387bbad3bdbe9040: Status 404 returned error can't find the container with id 076d4a3353e38ffd0f0632a16ba84af39e216a4f2c73a466387bbad3bdbe9040
Oct 30 00:51:53.118148 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:53.118114    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd" event={"ID":"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e","Type":"ContainerStarted","Data":"076d4a3353e38ffd0f0632a16ba84af39e216a4f2c73a466387bbad3bdbe9040"}
Oct 30 00:51:54.123441 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:54.123396    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd" event={"ID":"fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e","Type":"ContainerStarted","Data":"f899f3ed380f0de28f222da6641aa1042686e45af46881482c3efe03cd102692"}
Oct 30 00:51:54.123830 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:54.123498    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:51:54.159507 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:51:54.159388    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd" podStartSLOduration=1.313218091 podStartE2EDuration="2.159375389s" totalImagesPullingTime="846.157298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:54.15814513 +0000 UTC m=+555.705835627" watchObservedRunningTime="2026-10-30 00:51:54.159375389 +0000 UTC m=+555.707065884"
Oct 30 00:52:05.129010 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:52:05.128977    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-76ntd"
Oct 30 00:52:09.102750 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:52:09.102642    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:52:38.623914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:52:38.623886    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:52:38.627661 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:52:38.627638    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:52:38.633893 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:52:38.633864    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:52:38.637868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:52:38.637848    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:52:39.103222 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:52:39.103182    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:53:09.101234 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:53:09.101196    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:53:14.653866 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:53:14.653830    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:53:39.102606 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:53:39.102570    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:54:09.102600 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:54:09.102521    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:54:15.346409 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.346371    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"]
Oct 30 00:54:15.350716 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.350691    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.364603 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.364574    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 00:54:15.376127 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.376094    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 00:54:15.376278 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.376231    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xc4jj\""
Oct 30 00:54:15.376348 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.376280    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 00:54:15.378100 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.378075    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"]
Oct 30 00:54:15.449255 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.449219    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.449499 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.449265    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.449499 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.449345    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.449499 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.449382    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zns\" (UniqueName: \"kubernetes.io/projected/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-kube-api-access-s7zns\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.449674 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.449497    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.449674 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.449555    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550352 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550313    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550352 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550352    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zns\" (UniqueName: \"kubernetes.io/projected/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-kube-api-access-s7zns\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550632 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550380    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550632 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550511    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550632 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550625    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550681    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550847 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550809    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550909 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550892    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.550953 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.550937    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.552852 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.552828    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.553210 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.553192    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.568826 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.568798    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zns\" (UniqueName: \"kubernetes.io/projected/626e0a50-e9f7-4b0f-aa09-226cba9a0dfa-kube-api-access-s7zns\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z2ktb\" (UID: \"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.665707 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.665628    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:15.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.956901    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"]
Oct 30 00:54:15.960686 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:15.960656    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626e0a50_e9f7_4b0f_aa09_226cba9a0dfa.slice/crio-8db8e9cc3fc92c1cce779d35a6c4c9a37b2f82f084ca8820ee666ee4f64ea5b8 WatchSource:0}: Error finding container 8db8e9cc3fc92c1cce779d35a6c4c9a37b2f82f084ca8820ee666ee4f64ea5b8: Status 404 returned error can't find the container with id 8db8e9cc3fc92c1cce779d35a6c4c9a37b2f82f084ca8820ee666ee4f64ea5b8
Oct 30 00:54:15.962059 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:15.962042    5104 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:54:16.545394 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:16.545356    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:54:16.768310 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:16.768263    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb" event={"ID":"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa","Type":"ContainerStarted","Data":"8db8e9cc3fc92c1cce779d35a6c4c9a37b2f82f084ca8820ee666ee4f64ea5b8"}
Oct 30 00:54:17.777146 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:17.777109    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb" event={"ID":"626e0a50-e9f7-4b0f-aa09-226cba9a0dfa","Type":"ContainerStarted","Data":"c08dec5f8c9373b9018734a3c03c75457082d66bccc5c5afd744efc94b092924"}
Oct 30 00:54:17.813531 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:17.813482    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb" podStartSLOduration=1.641267112 podStartE2EDuration="2.813446006s" totalImagesPullingTime="1.172178894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:17.813043133 +0000 UTC m=+699.360733630" watchObservedRunningTime="2026-10-30 00:54:17.813446006 +0000 UTC m=+699.361136499"
Oct 30 00:54:18.666654 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:18.666603    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:19.550516 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.550472    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"]
Oct 30 00:54:19.554709 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.554686    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.581253 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.581220    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 00:54:19.582769 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.582741    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"]
Oct 30 00:54:19.585556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.585533    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d0232a-84d5-44ff-b2b6-d45f390c392d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.585672 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.585594    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8s8\" (UniqueName: \"kubernetes.io/projected/96d0232a-84d5-44ff-b2b6-d45f390c392d-kube-api-access-fm8s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.585751 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.585732    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.585791 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.585769    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.585827 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.585797    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.585885 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.585860    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.686992 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.686956    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d0232a-84d5-44ff-b2b6-d45f390c392d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687007    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8s8\" (UniqueName: \"kubernetes.io/projected/96d0232a-84d5-44ff-b2b6-d45f390c392d-kube-api-access-fm8s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687043    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687066    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687089    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687103    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687681 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687648    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687805 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687680    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.687805 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.687694    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.689484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.689427    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d0232a-84d5-44ff-b2b6-d45f390c392d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.689777 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.689753    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d0232a-84d5-44ff-b2b6-d45f390c392d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.705525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.705494    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8s8\" (UniqueName: \"kubernetes.io/projected/96d0232a-84d5-44ff-b2b6-d45f390c392d-kube-api-access-fm8s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9\" (UID: \"96d0232a-84d5-44ff-b2b6-d45f390c392d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:19.869922 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:19.869836    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:20.162932 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:20.162848    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"]
Oct 30 00:54:20.166298 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:20.166270    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d0232a_84d5_44ff_b2b6_d45f390c392d.slice/crio-ca1e8692de6636da0be2e4fe5c71537da97575ed87a67026e85a6f37ffcd6785 WatchSource:0}: Error finding container ca1e8692de6636da0be2e4fe5c71537da97575ed87a67026e85a6f37ffcd6785: Status 404 returned error can't find the container with id ca1e8692de6636da0be2e4fe5c71537da97575ed87a67026e85a6f37ffcd6785
Oct 30 00:54:20.791667 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:20.791629    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9" event={"ID":"96d0232a-84d5-44ff-b2b6-d45f390c392d","Type":"ContainerStarted","Data":"8c5a4c1f4977f5fd82e5910f7970c8af7d223ad2d58c219855d9ee4349bc7c1f"}
Oct 30 00:54:20.792096 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:20.791680    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9" event={"ID":"96d0232a-84d5-44ff-b2b6-d45f390c392d","Type":"ContainerStarted","Data":"ca1e8692de6636da0be2e4fe5c71537da97575ed87a67026e85a6f37ffcd6785"}
Oct 30 00:54:20.828445 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:20.828387    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9" podStartSLOduration=1.642291112 podStartE2EDuration="1.82837079s" totalImagesPullingTime="186.079678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:20.8276137 +0000 UTC m=+702.375304192" watchObservedRunningTime="2026-10-30 00:54:20.82837079 +0000 UTC m=+702.376061286"
Oct 30 00:54:20.870560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:20.870530    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:21.119064 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:21.118971    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:54:25.666142 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:25.666103    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:25.678824 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:25.678798    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:25.826493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:25.826448    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb"
Oct 30 00:54:27.245243 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.245208    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"]
Oct 30 00:54:27.249152 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.249126    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.267399 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.267372    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 00:54:27.280507 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.280473    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"]
Oct 30 00:54:27.359671 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.359636    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vqb\" (UniqueName: \"kubernetes.io/projected/955ad4ba-5e31-4758-8666-744d4549da0b-kube-api-access-l7vqb\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.359848 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.359681    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955ad4ba-5e31-4758-8666-744d4549da0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.359848 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.359765    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.359918 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.359870    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.359918 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.359896    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.359998 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.359985    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.407019 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.406969    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:54:27.461381 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461344    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955ad4ba-5e31-4758-8666-744d4549da0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.461581 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461427    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.461581 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461500    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.461581 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461526    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.461759 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461671    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.461816 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461802    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vqb\" (UniqueName: \"kubernetes.io/projected/955ad4ba-5e31-4758-8666-744d4549da0b-kube-api-access-l7vqb\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.461946 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461923    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.462029 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.461970    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.462086 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.462058    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.464171 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.464149    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955ad4ba-5e31-4758-8666-744d4549da0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.464428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.464407    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955ad4ba-5e31-4758-8666-744d4549da0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.480376 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.480343    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vqb\" (UniqueName: \"kubernetes.io/projected/955ad4ba-5e31-4758-8666-744d4549da0b-kube-api-access-l7vqb\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz\" (UID: \"955ad4ba-5e31-4758-8666-744d4549da0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.563000 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.562896    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:27.866531 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:27.866503    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"]
Oct 30 00:54:27.869217 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:27.869190    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955ad4ba_5e31_4758_8666_744d4549da0b.slice/crio-47e446a2bab61824580a3e7002d95e1c5c2a945a2e358f43da2721b5547856b4 WatchSource:0}: Error finding container 47e446a2bab61824580a3e7002d95e1c5c2a945a2e358f43da2721b5547856b4: Status 404 returned error can't find the container with id 47e446a2bab61824580a3e7002d95e1c5c2a945a2e358f43da2721b5547856b4
Oct 30 00:54:28.831290 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:28.831255    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz" event={"ID":"955ad4ba-5e31-4758-8666-744d4549da0b","Type":"ContainerStarted","Data":"f73cf66e5e157c77b415dc4121e4025a37ddb0490ccba5989653e9aa78c5dca0"}
Oct 30 00:54:28.831767 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:28.831312    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz" event={"ID":"955ad4ba-5e31-4758-8666-744d4549da0b","Type":"ContainerStarted","Data":"47e446a2bab61824580a3e7002d95e1c5c2a945a2e358f43da2721b5547856b4"}
Oct 30 00:54:28.870093 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:28.870017    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz" podStartSLOduration=1.688546187 podStartE2EDuration="1.870003522s" totalImagesPullingTime="181.457335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:28.868137518 +0000 UTC m=+710.415828013" watchObservedRunningTime="2026-10-30 00:54:28.870003522 +0000 UTC m=+710.417694018"
Oct 30 00:54:29.563513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:29.563447    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:29.870619 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:29.870528    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:29.883366 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:29.883337    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:30.852738 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:30.852709    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9"
Oct 30 00:54:32.444543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:32.444495    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:54:37.264269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.264235    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"]
Oct 30 00:54:37.268093 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.268061    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.282817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.282793    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5fml\""
Oct 30 00:54:37.296421 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.296392    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"]
Oct 30 00:54:37.356776 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.356737    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.356776 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.356783    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.357005 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.356800    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.357005 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.356927    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.357005 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.356947    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.357005 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.356980    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e840b-ae82-497a-9c10-e123a3e57e32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.357005 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.357000    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.357173 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.357062    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e840b-ae82-497a-9c10-e123a3e57e32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.357173 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.357100    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbf6k\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-kube-api-access-bbf6k\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.457945 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.457901    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e840b-ae82-497a-9c10-e123a3e57e32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.457945 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.457945    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.457964    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e840b-ae82-497a-9c10-e123a3e57e32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458000    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbf6k\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-kube-api-access-bbf6k\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458064    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458105    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458124    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458146    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458450 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458165    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458611 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458583    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458676 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458618    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458676 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458654    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458812 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458795    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e840b-ae82-497a-9c10-e123a3e57e32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.458860 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.458842    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.460526 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.460500    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.460672 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.460656    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e840b-ae82-497a-9c10-e123a3e57e32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.476036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.476013    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.476149 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.476051    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbf6k\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-kube-api-access-bbf6k\") pod \"maas-default-gateway-openshift-default-67bf986f99-bq26k\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.563671 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.563574    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:37.576565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.576536    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:37.581034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.581013    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:37.884515 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.884419    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz"
Oct 30 00:54:37.888298 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:37.888269    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"]
Oct 30 00:54:37.890983 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:37.890955    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103e840b_ae82_497a_9c10_e123a3e57e32.slice/crio-e1e316187634f732af6359a99f49c13a330cc26fb25ada8b09bb40c9b831d3b2 WatchSource:0}: Error finding container e1e316187634f732af6359a99f49c13a330cc26fb25ada8b09bb40c9b831d3b2: Status 404 returned error can't find the container with id e1e316187634f732af6359a99f49c13a330cc26fb25ada8b09bb40c9b831d3b2
Oct 30 00:54:38.878924 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:38.878881    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" event={"ID":"103e840b-ae82-497a-9c10-e123a3e57e32","Type":"ContainerStarted","Data":"e1e316187634f732af6359a99f49c13a330cc26fb25ada8b09bb40c9b831d3b2"}
Oct 30 00:54:39.106765 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:54:39.106722    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:54:40.287508 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:40.287476    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:40.287838 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:40.287547    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:40.287838 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:40.287577    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:40.904004 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:40.903965    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" event={"ID":"103e840b-ae82-497a-9c10-e123a3e57e32","Type":"ContainerStarted","Data":"8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f"}
Oct 30 00:54:40.942786 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:40.942737    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podStartSLOduration=1.549181657 podStartE2EDuration="3.942724399s" totalImagesPullingTime="2.393542742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:40.941434554 +0000 UTC m=+722.489125079" watchObservedRunningTime="2026-10-30 00:54:40.942724399 +0000 UTC m=+722.490414895"
Oct 30 00:54:41.589482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:41.588867    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:41.597484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:41.596592    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:54:41.597484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:41.596678    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:54:42.595482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:42.589591    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:54:42.595482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:42.589691    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:54:43.595621 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:43.595575    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:54:43.596196 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:43.596166    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:54:44.297336 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.297291    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-nbnzs"]
Oct 30 00:54:44.322550 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.322350    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-nbnzs"]
Oct 30 00:54:44.322799 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.322534    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.336776 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.336730    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 00:54:44.336911 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.336881    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-glljh\""
Oct 30 00:54:44.336976 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.336913    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 00:54:44.430517 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.430443    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a0ef49-c543-4f0d-925e-efd31a5e1272-metrics-certs\") pod \"ai-gateway-controller-655fc879db-nbnzs\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") " pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.430680 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.430573    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87z7\" (UniqueName: \"kubernetes.io/projected/08a0ef49-c543-4f0d-925e-efd31a5e1272-kube-api-access-t87z7\") pod \"ai-gateway-controller-655fc879db-nbnzs\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") " pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.532103 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.532062    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a0ef49-c543-4f0d-925e-efd31a5e1272-metrics-certs\") pod \"ai-gateway-controller-655fc879db-nbnzs\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") " pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.532306 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.532115    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87z7\" (UniqueName: \"kubernetes.io/projected/08a0ef49-c543-4f0d-925e-efd31a5e1272-kube-api-access-t87z7\") pod \"ai-gateway-controller-655fc879db-nbnzs\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") " pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.534706 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.534678    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a0ef49-c543-4f0d-925e-efd31a5e1272-metrics-certs\") pod \"ai-gateway-controller-655fc879db-nbnzs\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") " pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.551746 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.551673    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87z7\" (UniqueName: \"kubernetes.io/projected/08a0ef49-c543-4f0d-925e-efd31a5e1272-kube-api-access-t87z7\") pod \"ai-gateway-controller-655fc879db-nbnzs\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") " pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.581911 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.581872    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:54:44.582191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.582159    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:54:44.641270 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.641228    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:44.999994 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:44.999966    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-nbnzs"]
Oct 30 00:54:45.002584 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:45.002550    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a0ef49_c543_4f0d_925e_efd31a5e1272.slice/crio-2977e66a68d132145fcd6362324378027f6fb10f5120f8cac58538d643c03713 WatchSource:0}: Error finding container 2977e66a68d132145fcd6362324378027f6fb10f5120f8cac58538d643c03713: Status 404 returned error can't find the container with id 2977e66a68d132145fcd6362324378027f6fb10f5120f8cac58538d643c03713
Oct 30 00:54:45.595510 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:45.595008    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:54:45.595510 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:45.595073    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:54:45.936667 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:45.936573    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs" event={"ID":"08a0ef49-c543-4f0d-925e-efd31a5e1272","Type":"ContainerStarted","Data":"2977e66a68d132145fcd6362324378027f6fb10f5120f8cac58538d643c03713"}
Oct 30 00:54:46.582285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:46.582253    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:54:46.582499 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:46.582310    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:54:47.581838 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:47.581795    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:54:47.582335 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:47.581859    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:54:47.949638 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:47.949605    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs" event={"ID":"08a0ef49-c543-4f0d-925e-efd31a5e1272","Type":"ContainerStarted","Data":"523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe"}
Oct 30 00:54:47.949850 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:47.949727    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:47.986820 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:47.986759    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs" podStartSLOduration=1.318072515 podStartE2EDuration="3.986738488s" totalImagesPullingTime="2.668665973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:47.985372792 +0000 UTC m=+729.533063289" watchObservedRunningTime="2026-10-30 00:54:47.986738488 +0000 UTC m=+729.534428984"
Oct 30 00:54:48.959609 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:48.959567    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:48.960771 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:48.960754    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:48.961612 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:48.961592    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 00:54:49.079704 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.079665    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"]
Oct 30 00:54:49.093028 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.092998    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.107007 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.106970    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"]
Oct 30 00:54:49.107308 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.107289    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 00:54:49.108101 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.108077    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9fnng\""
Oct 30 00:54:49.178057 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.178018    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-txfcv\" (UID: \"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.178257 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.178073    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz4h7\" (UniqueName: \"kubernetes.io/projected/9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae-kube-api-access-xz4h7\") pod \"payload-pre-processing-7b98bb678c-txfcv\" (UID: \"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.279061 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.278954    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-txfcv\" (UID: \"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.279061 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.279007    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz4h7\" (UniqueName: \"kubernetes.io/projected/9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae-kube-api-access-xz4h7\") pod \"payload-pre-processing-7b98bb678c-txfcv\" (UID: \"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.279728 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.279707    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-txfcv\" (UID: \"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.298361 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.298329    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz4h7\" (UniqueName: \"kubernetes.io/projected/9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae-kube-api-access-xz4h7\") pod \"payload-pre-processing-7b98bb678c-txfcv\" (UID: \"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.409428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.409389    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:49.716699 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.716665    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"]
Oct 30 00:54:49.719069 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:49.719034    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0ad3ff_bb3d_4c7f_b9aa_f9a89c9b59ae.slice/crio-de733ce68c5681325bfdd5fa43fef32f42f1198c80c2e68a3a7b3f35212c493b WatchSource:0}: Error finding container de733ce68c5681325bfdd5fa43fef32f42f1198c80c2e68a3a7b3f35212c493b: Status 404 returned error can't find the container with id de733ce68c5681325bfdd5fa43fef32f42f1198c80c2e68a3a7b3f35212c493b
Oct 30 00:54:49.972208 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:49.972114    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv" event={"ID":"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae","Type":"ContainerStarted","Data":"de733ce68c5681325bfdd5fa43fef32f42f1198c80c2e68a3a7b3f35212c493b"}
Oct 30 00:54:52.319075 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.318990    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"]
Oct 30 00:54:52.322766 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.322744    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.350643 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.350609    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"]
Oct 30 00:54:52.411621 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411581    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.411621 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411629    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.411891 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411719    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.411891 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411751    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8wg\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-kube-api-access-st8wg\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.411891 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411794    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.411891 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411814    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.412097 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411941    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca08310-2783-4282-be94-3d3a1da31f23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.412097 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.411993    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca08310-2783-4282-be94-3d3a1da31f23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.412097 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.412050    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.512856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.512821    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.512856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.512860    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.512893    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.512909    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8wg\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-kube-api-access-st8wg\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.512946    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.512962    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.512992    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca08310-2783-4282-be94-3d3a1da31f23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.513036    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca08310-2783-4282-be94-3d3a1da31f23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.513070    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513425 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.513376    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.513511    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513667 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.513573    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513667 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.513643    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.513846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.513826    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca08310-2783-4282-be94-3d3a1da31f23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.515528 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.515502    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.515651 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.515634    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca08310-2783-4282-be94-3d3a1da31f23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.530986 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.530947    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.531185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.531161    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8wg\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-kube-api-access-st8wg\") pod \"maas-default-gateway-openshift-default-67bf986f99-fj9f6\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.635337 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.635247    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:52.947606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.947538    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"]
Oct 30 00:54:52.949980 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:52.949948    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca08310_2783_4282_be94_3d3a1da31f23.slice/crio-17aa689439ca1ad57ed346ebd77123082ece485a7a998860766cf460d8e84c26 WatchSource:0}: Error finding container 17aa689439ca1ad57ed346ebd77123082ece485a7a998860766cf460d8e84c26: Status 404 returned error can't find the container with id 17aa689439ca1ad57ed346ebd77123082ece485a7a998860766cf460d8e84c26
Oct 30 00:54:52.953352 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.953307    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:52.953440 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.953401    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:52.953517 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.953445    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:54:52.990502 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.990453    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" event={"ID":"5ca08310-2783-4282-be94-3d3a1da31f23","Type":"ContainerStarted","Data":"17aa689439ca1ad57ed346ebd77123082ece485a7a998860766cf460d8e84c26"}
Oct 30 00:54:52.992110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.992083    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv" event={"ID":"9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae","Type":"ContainerStarted","Data":"063ed1fa872adb01b979c11869525825f0a73ce545bfae29b2477739286f24e6"}
Oct 30 00:54:52.992236 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:52.992224    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:54:53.029994 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:53.029942    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv" podStartSLOduration=1.8136547859999999 podStartE2EDuration="4.02992763s" totalImagesPullingTime="2.216272844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:53.028172718 +0000 UTC m=+734.575863213" watchObservedRunningTime="2026-10-30 00:54:53.02992763 +0000 UTC m=+734.577618126"
Oct 30 00:54:54.003131 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:54.003092    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" event={"ID":"5ca08310-2783-4282-be94-3d3a1da31f23","Type":"ContainerStarted","Data":"e27347681759eb1a84bfa8ac5ad90731647c6da1ca7dd3fcea00eed0af95ec53"}
Oct 30 00:54:54.636111 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:54.636073    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:54:54.637757 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:54.637730    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:54:54.637863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:54.637781    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:54:55.636411 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:55.636383    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:54:55.636812 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:55.636432    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:54:56.636136 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:56.636102    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:54:56.636311 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:56.636155    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:54:57.636409 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:57.636368    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:54:57.636796 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:57.636430    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:54:58.636174 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:58.636137    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:54:58.636338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:58.636192    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:54:58.961450 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:58.961416    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:54:58.997563 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:58.997507    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podStartSLOduration=6.997489734 podStartE2EDuration="6.997489734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:54.063831979 +0000 UTC m=+735.611522474" watchObservedRunningTime="2026-10-30 00:54:58.997489734 +0000 UTC m=+740.545180232"
Oct 30 00:54:59.247672 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.247562    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6f85488d-567cx"]
Oct 30 00:54:59.254219 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.254181    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.269028 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.268998    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6f85488d-567cx"]
Oct 30 00:54:59.385997 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.385962    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qf4\" (UniqueName: \"kubernetes.io/projected/f4a8486d-468f-404b-b9cc-67d667467964-kube-api-access-s4qf4\") pod \"ai-gateway-controller-7d6f85488d-567cx\" (UID: \"f4a8486d-468f-404b-b9cc-67d667467964\") " pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.386145 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.386014    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a8486d-468f-404b-b9cc-67d667467964-metrics-certs\") pod \"ai-gateway-controller-7d6f85488d-567cx\" (UID: \"f4a8486d-468f-404b-b9cc-67d667467964\") " pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.486863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.486829    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qf4\" (UniqueName: \"kubernetes.io/projected/f4a8486d-468f-404b-b9cc-67d667467964-kube-api-access-s4qf4\") pod \"ai-gateway-controller-7d6f85488d-567cx\" (UID: \"f4a8486d-468f-404b-b9cc-67d667467964\") " pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.487037 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.486877    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a8486d-468f-404b-b9cc-67d667467964-metrics-certs\") pod \"ai-gateway-controller-7d6f85488d-567cx\" (UID: \"f4a8486d-468f-404b-b9cc-67d667467964\") " pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.489533 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.489500    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a8486d-468f-404b-b9cc-67d667467964-metrics-certs\") pod \"ai-gateway-controller-7d6f85488d-567cx\" (UID: \"f4a8486d-468f-404b-b9cc-67d667467964\") " pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.505834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.505764    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qf4\" (UniqueName: \"kubernetes.io/projected/f4a8486d-468f-404b-b9cc-67d667467964-kube-api-access-s4qf4\") pod \"ai-gateway-controller-7d6f85488d-567cx\" (UID: \"f4a8486d-468f-404b-b9cc-67d667467964\") " pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.568854 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.568815    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:54:59.636126 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.636091    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:54:59.636254 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.636178    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 00:54:59.866628 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:54:59.866594    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6f85488d-567cx"]
Oct 30 00:54:59.869645 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:54:59.869616    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a8486d_468f_404b_b9cc_67d667467964.slice/crio-df81adedafa77c288b87129b8707e5a0d8d5463fc77de89c2a3bcd8235123b50 WatchSource:0}: Error finding container df81adedafa77c288b87129b8707e5a0d8d5463fc77de89c2a3bcd8235123b50: Status 404 returned error can't find the container with id df81adedafa77c288b87129b8707e5a0d8d5463fc77de89c2a3bcd8235123b50
Oct 30 00:55:00.035976 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:00.035943    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx" event={"ID":"f4a8486d-468f-404b-b9cc-67d667467964","Type":"ContainerStarted","Data":"df81adedafa77c288b87129b8707e5a0d8d5463fc77de89c2a3bcd8235123b50"}
Oct 30 00:55:01.043751 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:01.043714    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx" event={"ID":"f4a8486d-468f-404b-b9cc-67d667467964","Type":"ContainerStarted","Data":"46c87bb65013d8f3894835d510c5a80cb996784dcd720f44b27b9a3df140fde9"}
Oct 30 00:55:01.044222 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:01.043783    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:55:01.080855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:01.080785    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx" podStartSLOduration=1.757359574 podStartE2EDuration="2.080764261s" totalImagesPullingTime="323.404687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:01.079197431 +0000 UTC m=+742.626887929" watchObservedRunningTime="2026-10-30 00:55:01.080764261 +0000 UTC m=+742.628454758"
Oct 30 00:55:01.375724 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:01.375632    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:55:01.376036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:01.376005    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:55:01.376831 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:01.376809    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 00:55:04.006700 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:04.006664    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-txfcv"
Oct 30 00:55:07.373213 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.373175    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"]
Oct 30 00:55:07.377125 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.377108    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.407623 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.407582    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"]
Oct 30 00:55:07.568277 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568242    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee3fb42-b55e-4887-b109-efbe113cef93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568445 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568358    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568445 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568411    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee3fb42-b55e-4887-b109-efbe113cef93-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568445 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568435    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568687 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568520    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568687 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568607    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee3fb42-b55e-4887-b109-efbe113cef93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568687 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568625    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdw2\" (UniqueName: \"kubernetes.io/projected/cee3fb42-b55e-4887-b109-efbe113cef93-kube-api-access-mfdw2\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568828 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568689    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.568828 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.568730    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.669837 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.669807    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee3fb42-b55e-4887-b109-efbe113cef93-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.669989 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.669848    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.669989 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.669966    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670102 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670004    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee3fb42-b55e-4887-b109-efbe113cef93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670102 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670030    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdw2\" (UniqueName: \"kubernetes.io/projected/cee3fb42-b55e-4887-b109-efbe113cef93-kube-api-access-mfdw2\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670102 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670064    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670275 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670123    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670356 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670328    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee3fb42-b55e-4887-b109-efbe113cef93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670616 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670481    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670616 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670520    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670616 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670485    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670651    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.670856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.670784    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.671045 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.671021    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee3fb42-b55e-4887-b109-efbe113cef93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.672517 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.672492    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee3fb42-b55e-4887-b109-efbe113cef93-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.672690 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.672673    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee3fb42-b55e-4887-b109-efbe113cef93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.687716 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.687692    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee3fb42-b55e-4887-b109-efbe113cef93-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.687873 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.687854    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdw2\" (UniqueName: \"kubernetes.io/projected/cee3fb42-b55e-4887-b109-efbe113cef93-kube-api-access-mfdw2\") pod \"maas-default-gateway-openshift-default-67bf986f99-j726m\" (UID: \"cee3fb42-b55e-4887-b109-efbe113cef93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.691651 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.691632    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:07.996451 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:07.996416    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"]
Oct 30 00:55:07.999143 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:07.999109    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee3fb42_b55e_4887_b109_efbe113cef93.slice/crio-9ca95d746ef3d0a480197928a875a810bb1dc54f7c03f80451e4d0d2ee6a64ca WatchSource:0}: Error finding container 9ca95d746ef3d0a480197928a875a810bb1dc54f7c03f80451e4d0d2ee6a64ca: Status 404 returned error can't find the container with id 9ca95d746ef3d0a480197928a875a810bb1dc54f7c03f80451e4d0d2ee6a64ca
Oct 30 00:55:08.002426 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:08.002395    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:08.002523 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:08.002481    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:08.002523 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:08.002509    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:08.079618 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:08.079578    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" event={"ID":"cee3fb42-b55e-4887-b109-efbe113cef93","Type":"ContainerStarted","Data":"450a921879c086862145e0a44dd5b480f18dfd9cec851356cdcad45c8255fadd"}
Oct 30 00:55:08.079778 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:08.079636    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" event={"ID":"cee3fb42-b55e-4887-b109-efbe113cef93","Type":"ContainerStarted","Data":"9ca95d746ef3d0a480197928a875a810bb1dc54f7c03f80451e4d0d2ee6a64ca"}
Oct 30 00:55:08.118526 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:08.118446    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podStartSLOduration=1.118431623 podStartE2EDuration="1.118431623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:08.116954329 +0000 UTC m=+749.664644826" watchObservedRunningTime="2026-10-30 00:55:08.118431623 +0000 UTC m=+749.666122119"
Oct 30 00:55:08.694481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:08.693802    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:09.125227 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:55:09.125140    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:55:09.698652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:09.698609    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:55:09.699150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:09.698679    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 00:55:10.697884 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:10.697843    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:55:10.698109 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:10.697945    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 00:55:11.696481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:11.693577    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:55:11.696481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:11.693683    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 00:55:12.050359 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.050262    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6f85488d-567cx"
Oct 30 00:55:12.110938 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.110901    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-nbnzs"]
Oct 30 00:55:12.111166 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.111117    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs" podUID="08a0ef49-c543-4f0d-925e-efd31a5e1272" containerName="manager" containerID="cri-o://523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe" gracePeriod=30
Oct 30 00:55:12.415499 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.415451    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:55:12.435363 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.435337    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a0ef49-c543-4f0d-925e-efd31a5e1272-kube-api-access-t87z7\" (UniqueName: \"kubernetes.io/projected/08a0ef49-c543-4f0d-925e-efd31a5e1272-kube-api-access-t87z7\") pod \"08a0ef49-c543-4f0d-925e-efd31a5e1272\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") "
Oct 30 00:55:12.435556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.435508    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a0ef49-c543-4f0d-925e-efd31a5e1272-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a0ef49-c543-4f0d-925e-efd31a5e1272-metrics-certs\") pod \"08a0ef49-c543-4f0d-925e-efd31a5e1272\" (UID: \"08a0ef49-c543-4f0d-925e-efd31a5e1272\") "
Oct 30 00:55:12.437602 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.437573    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a0ef49-c543-4f0d-925e-efd31a5e1272-metrics-certs" pod "08a0ef49-c543-4f0d-925e-efd31a5e1272" (UID: "08a0ef49-c543-4f0d-925e-efd31a5e1272"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:55:12.437723 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.437703    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a0ef49-c543-4f0d-925e-efd31a5e1272-kube-api-access-t87z7" pod "08a0ef49-c543-4f0d-925e-efd31a5e1272" (UID: "08a0ef49-c543-4f0d-925e-efd31a5e1272"). InnerVolumeSpecName "kube-api-access-t87z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:12.536498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.536439    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08a0ef49-c543-4f0d-925e-efd31a5e1272-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:55:12.536498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.536496    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87z7\" (UniqueName: \"kubernetes.io/projected/08a0ef49-c543-4f0d-925e-efd31a5e1272-kube-api-access-t87z7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:55:12.697613 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.697573    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:55:12.698108 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:12.697639    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 00:55:13.130945 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.130913    5104 generic.go:356] "Generic (PLEG): container finished" podID="08a0ef49-c543-4f0d-925e-efd31a5e1272" containerID="523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe" exitCode=0
Oct 30 00:55:13.131119 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.130956    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs" event={"ID":"08a0ef49-c543-4f0d-925e-efd31a5e1272","Type":"ContainerDied","Data":"523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe"}
Oct 30 00:55:13.131119 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.130962    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs"
Oct 30 00:55:13.131119 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.131002    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-nbnzs" event={"ID":"08a0ef49-c543-4f0d-925e-efd31a5e1272","Type":"ContainerDied","Data":"2977e66a68d132145fcd6362324378027f6fb10f5120f8cac58538d643c03713"}
Oct 30 00:55:13.131119 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.131026    5104 scope.go:122] "RemoveContainer" containerID="523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe"
Oct 30 00:55:13.140332 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.140310    5104 scope.go:122] "RemoveContainer" containerID="523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe"
Oct 30 00:55:13.140675 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:55:13.140654    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe\": container with ID starting with 523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe not found: ID does not exist" containerID="523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe"
Oct 30 00:55:13.140764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.140683    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe"} err="failed to get container status \"523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe\": rpc error: code = NotFound desc = could not find container \"523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe\": container with ID starting with 523f19bf346aeb3803c058cc9c7eb60c8ca6f8f517439702de75350db886d9fe not found: ID does not exist"
Oct 30 00:55:13.197552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.192014    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-nbnzs"]
Oct 30 00:55:13.203111 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.203078    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-nbnzs"]
Oct 30 00:55:13.694927 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.694882    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:55:13.695247 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:13.695213    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 00:55:14.576699 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:14.576667    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a0ef49-c543-4f0d-925e-efd31a5e1272" path="/var/lib/kubelet/pods/08a0ef49-c543-4f0d-925e-efd31a5e1272/volumes"
Oct 30 00:55:14.692765 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:14.692732    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:55:14.692952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:14.692786    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 00:55:16.305433 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:16.305400    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:16.305895 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:16.305778    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:16.306609 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:16.306587    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m"
Oct 30 00:55:22.419700 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.419667    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"]
Oct 30 00:55:22.420186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.420170    5104 cpu_manager.go:414] "containerMap: removing container" podUID="08a0ef49-c543-4f0d-925e-efd31a5e1272" containerName="manager"
Oct 30 00:55:22.420186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.420186    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a0ef49-c543-4f0d-925e-efd31a5e1272" containerName="manager"
Oct 30 00:55:22.420320 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.420278    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a0ef49-c543-4f0d-925e-efd31a5e1272" containerName="manager"
Oct 30 00:55:22.425494 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.425473    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.456600 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.456564    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"]
Oct 30 00:55:22.537545 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537493    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aeb239-a767-4b4c-9263-e28d77d951bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537545 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537544    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537809 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537675    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537809 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537753    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537879 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537813    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537879 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537844    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfvw\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-kube-api-access-hpfvw\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537879 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537866    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537989 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537883    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.537989 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.537932    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639412 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639375    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aeb239-a767-4b4c-9263-e28d77d951bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639412 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639415    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639481    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639522    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639545    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639562    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfvw\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-kube-api-access-hpfvw\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639597    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639616    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.639992 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.639897    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.640075 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.640043    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.640129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.640100    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.640185 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.640165    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.640266 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.640251    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.640301 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.640268    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aeb239-a767-4b4c-9263-e28d77d951bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.642610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.642584    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.642740 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.642724    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.657778 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.657744    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.657914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.657882    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfvw\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-kube-api-access-hpfvw\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbh6w\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:22.740323 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:22.740239    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:23.052866 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.048870    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"]
Oct 30 00:55:23.054572 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:23.054540    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50aeb239_a767_4b4c_9263_e28d77d951bd.slice/crio-f8fd9fc880a40d8422771d946abab7a092485acabce83942a2ac78de469e44aa WatchSource:0}: Error finding container f8fd9fc880a40d8422771d946abab7a092485acabce83942a2ac78de469e44aa: Status 404 returned error can't find the container with id f8fd9fc880a40d8422771d946abab7a092485acabce83942a2ac78de469e44aa
Oct 30 00:55:23.056610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.056579    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:23.056684 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.056643    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:23.056684 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.056669    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:23.185410 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.185367    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" event={"ID":"50aeb239-a767-4b4c-9263-e28d77d951bd","Type":"ContainerStarted","Data":"df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a"}
Oct 30 00:55:23.185637 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.185436    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" event={"ID":"50aeb239-a767-4b4c-9263-e28d77d951bd","Type":"ContainerStarted","Data":"f8fd9fc880a40d8422771d946abab7a092485acabce83942a2ac78de469e44aa"}
Oct 30 00:55:23.224053 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.223928    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podStartSLOduration=1.223908112 podStartE2EDuration="1.223908112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:23.223366599 +0000 UTC m=+764.771057107" watchObservedRunningTime="2026-10-30 00:55:23.223908112 +0000 UTC m=+764.771598611"
Oct 30 00:55:23.740582 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:23.740543    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:24.742838 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:24.742803    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:55:24.743254 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:24.742856    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:55:25.740924 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:25.740884    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:55:25.741129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:25.740937    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:55:26.741086 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:26.741054    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:55:26.741515 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:26.741105    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:55:27.741352 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:27.741318    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:55:27.741775 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:27.741372    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:55:28.741099 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:28.741059    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:55:28.741298 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:28.741119    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:55:29.741510 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:29.741405    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:55:29.741510 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:29.741482    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 00:55:31.399876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:31.399845    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:31.400355 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:31.400212    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:31.400980 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:31.400958    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 00:55:39.104375 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:55:39.104288    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:55:50.045315 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.045281    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"]
Oct 30 00:55:50.053574 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.053544    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.071335 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.071307    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dca38d-openshift-default-dockercfg-b9zlp\""
Oct 30 00:55:50.084565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.084533    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"]
Oct 30 00:55:50.113006 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.112962    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113174 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113026    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113215 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113167    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113250 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113226    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113288 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113261    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113322 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113294    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czx5\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-kube-api-access-6czx5\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113366 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113350    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65623efa-d0c0-455c-a784-3b48e1e9eb24-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113450    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.113529 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.113515    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214270 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214231    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214270 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214278    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214598 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214302    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214598 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214330    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214598 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214386    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214598 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214497    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214598 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214558    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214896 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214592    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czx5\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-kube-api-access-6czx5\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.214896 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.214673    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65623efa-d0c0-455c-a784-3b48e1e9eb24-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.215105 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.215075    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.215504 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.215476    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.215636 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.215503    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65623efa-d0c0-455c-a784-3b48e1e9eb24-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.215985 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.215951    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.216110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.215984    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.217334 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.217305    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.217646 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.217627    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.249304 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.249268    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.258110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.258077    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czx5\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-kube-api-access-6czx5\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.376521 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.376429    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:50.520247 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.520202    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"]
Oct 30 00:55:50.526666 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.526635    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.545058 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.544998    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d827b7-openshift-default-dockercfg-95vfg\""
Oct 30 00:55:50.548232 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.548200    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"]
Oct 30 00:55:50.622388 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622345    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622597 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622394    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622597 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622517    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622597 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622553    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8p28\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-kube-api-access-r8p28\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622597 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622575    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622743 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622720    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622780 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622755    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622813 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622777    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.622976 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.622957    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.670540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.669967    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"]
Oct 30 00:55:50.675197 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.675175    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.690612 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.690572    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c4847c-openshift-default-dockercfg-4nf8k\""
Oct 30 00:55:50.695928 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.695864    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"]
Oct 30 00:55:50.702953 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.702923    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"]
Oct 30 00:55:50.707814 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:50.707781    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65623efa_d0c0_455c_a784_3b48e1e9eb24.slice/crio-4b8e1a23da529ec6fc242c445e919d5cee3f53f5580ea8a7fe2dce5580ec5c83 WatchSource:0}: Error finding container 4b8e1a23da529ec6fc242c445e919d5cee3f53f5580ea8a7fe2dce5580ec5c83: Status 404 returned error can't find the container with id 4b8e1a23da529ec6fc242c445e919d5cee3f53f5580ea8a7fe2dce5580ec5c83
Oct 30 00:55:50.710113 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.710074    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:50.710245 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.710162    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:50.710245 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.710230    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:50.725272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.725234    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.725448 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.725318    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.725567 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.725479    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8p28\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-kube-api-access-r8p28\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.725657 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.725639    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.725711 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.725677    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.726016 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.725937    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tcwz\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-kube-api-access-9tcwz\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.726159 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.726094    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17386f66-3f21-4f9e-8a12-9700848daaf9-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.726244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.726223    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.727644 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.727619    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.728261 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728239    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.728358 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728290    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.728358 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728338    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.728503 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728374    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.728503 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728403    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.728503 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728466    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.728693 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728500    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.728693 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728539    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.728693 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728571    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.728693 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728596    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.728693 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.728623    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.729589 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.729566    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.730112 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.730089    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.730383 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.730362    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.731878 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.731847    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.731994 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.731848    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.747216 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.747186    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.748099 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.748075    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8p28\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-kube-api-access-r8p28\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.829758 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829709    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.829965 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829773    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcwz\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-kube-api-access-9tcwz\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.829965 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829817    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17386f66-3f21-4f9e-8a12-9700848daaf9-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.829965 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829852    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.829965 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829888    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.829965 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829926    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.829965 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829962    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.830300 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.829989    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.830300 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.830040    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.831203 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.831172    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.833470 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.832109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17386f66-3f21-4f9e-8a12-9700848daaf9-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.833470 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.832934    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.834055 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.834011    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.834401 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.834383    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.840087 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.840041    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.845922 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.845891    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:50.847325 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.847303    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.879952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.859108    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcwz\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-kube-api-access-9tcwz\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.879952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.868081    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:50.970616 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.963071    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"]
Oct 30 00:55:50.979226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.976886    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"]
Oct 30 00:55:50.979226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.977024    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:50.993563 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.992967    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fd3281-openshift-default-dockercfg-7q6pm\""
Oct 30 00:55:50.993717 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:50.993683    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:51.033525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033484    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.033845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033574    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.033845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033645    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.033845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033678    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.033845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033716    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.033845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033780    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.033845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033817    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.034182 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033846    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmkb\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-kube-api-access-qpmkb\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.034182 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.033887    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154279    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154356    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154432    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154511    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154544    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154582    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154656    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154693    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.154727    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmkb\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-kube-api-access-qpmkb\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.155702    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.156372    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.160282    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.160586    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.162513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.160851    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.164096 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.164041    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.183845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.182260    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.235863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.213815    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmkb\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-kube-api-access-qpmkb\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.235863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.218605    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.235863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.220391    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"]
Oct 30 00:55:51.235863 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:51.227624    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03188ad_e1df_4bdf_bcac_b9d46a5e704b.slice/crio-219096a11fb0ff4c61e43afd73606b1d2e3186dc066b816edf3acb0a287f08c8 WatchSource:0}: Error finding container 219096a11fb0ff4c61e43afd73606b1d2e3186dc066b816edf3acb0a287f08c8: Status 404 returned error can't find the container with id 219096a11fb0ff4c61e43afd73606b1d2e3186dc066b816edf3acb0a287f08c8
Oct 30 00:55:51.235863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.231983    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.235863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.232076    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.235863 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.232117    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.297295 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.297264    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:51.325127 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.325101    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"]
Oct 30 00:55:51.332312 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:51.332283    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17386f66_3f21_4f9e_8a12_9700848daaf9.slice/crio-afc2d96faf7794072ac3706eb0181045e8b7dc2a6aa1c75c20615b9d7649ab05 WatchSource:0}: Error finding container afc2d96faf7794072ac3706eb0181045e8b7dc2a6aa1c75c20615b9d7649ab05: Status 404 returned error can't find the container with id afc2d96faf7794072ac3706eb0181045e8b7dc2a6aa1c75c20615b9d7649ab05
Oct 30 00:55:51.336352 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.336311    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.336451 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.336406    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.336545 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.336490    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.339156 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.339130    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" event={"ID":"17386f66-3f21-4f9e-8a12-9700848daaf9","Type":"ContainerStarted","Data":"afc2d96faf7794072ac3706eb0181045e8b7dc2a6aa1c75c20615b9d7649ab05"}
Oct 30 00:55:51.340804 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.340785    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" event={"ID":"d03188ad-e1df-4bdf-bcac-b9d46a5e704b","Type":"ContainerStarted","Data":"219096a11fb0ff4c61e43afd73606b1d2e3186dc066b816edf3acb0a287f08c8"}
Oct 30 00:55:51.342338 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.342313    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" event={"ID":"65623efa-d0c0-455c-a784-3b48e1e9eb24","Type":"ContainerStarted","Data":"c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a"}
Oct 30 00:55:51.342428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.342356    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" event={"ID":"65623efa-d0c0-455c-a784-3b48e1e9eb24","Type":"ContainerStarted","Data":"4b8e1a23da529ec6fc242c445e919d5cee3f53f5580ea8a7fe2dce5580ec5c83"}
Oct 30 00:55:51.380660 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.380055    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:51.397470 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.396579    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podStartSLOduration=1.396560023 podStartE2EDuration="1.396560023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:51.395881904 +0000 UTC m=+792.943572427" watchObservedRunningTime="2026-10-30 00:55:51.396560023 +0000 UTC m=+792.944250521"
Oct 30 00:55:51.399382 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.399362    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:51.612577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.612483    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"]
Oct 30 00:55:51.615669 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:51.615611    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfedb57_fd1c_4d6f_9d27_249f3a7f244a.slice/crio-e0d2c9db3d66c8640753691dc61dea3f454b599db91f3e10e7b83dbce7943b41 WatchSource:0}: Error finding container e0d2c9db3d66c8640753691dc61dea3f454b599db91f3e10e7b83dbce7943b41: Status 404 returned error can't find the container with id e0d2c9db3d66c8640753691dc61dea3f454b599db91f3e10e7b83dbce7943b41
Oct 30 00:55:51.618563 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.618517    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.618673 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.618577    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:51.618673 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:51.618610    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:55:52.348606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.348566    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" event={"ID":"17386f66-3f21-4f9e-8a12-9700848daaf9","Type":"ContainerStarted","Data":"264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a"}
Oct 30 00:55:52.350281 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.350250    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" event={"ID":"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a","Type":"ContainerStarted","Data":"5e1d3247a02de3ab22ef77909c2b6b5258322e042ad4492576e5842b4f139477"}
Oct 30 00:55:52.350416 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.350295    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" event={"ID":"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a","Type":"ContainerStarted","Data":"e0d2c9db3d66c8640753691dc61dea3f454b599db91f3e10e7b83dbce7943b41"}
Oct 30 00:55:52.351905 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.351881    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" event={"ID":"d03188ad-e1df-4bdf-bcac-b9d46a5e704b","Type":"ContainerStarted","Data":"0a0bd83d509ee3c6cf4f080b5691f1eb6f9be5bdf00bbf00fd59b88ec95eaf25"}
Oct 30 00:55:52.352078 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.352061    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:52.353031 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.353003    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:55:52.416342 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.416278    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podStartSLOduration=2.416258987 podStartE2EDuration="2.416258987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:52.401647105 +0000 UTC m=+793.949337602" watchObservedRunningTime="2026-10-30 00:55:52.416258987 +0000 UTC m=+793.963949483"
Oct 30 00:55:52.441124 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.441068    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podStartSLOduration=2.441056701 podStartE2EDuration="2.441056701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:52.439632766 +0000 UTC m=+793.987323262" watchObservedRunningTime="2026-10-30 00:55:52.441056701 +0000 UTC m=+793.988747197"
Oct 30 00:55:52.525916 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.525852    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podStartSLOduration=2.525831316 podStartE2EDuration="2.525831316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:52.52447429 +0000 UTC m=+794.072164783" watchObservedRunningTime="2026-10-30 00:55:52.525831316 +0000 UTC m=+794.073521812"
Oct 30 00:55:52.848407 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.848372    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:52.854737 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.854711    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:52.994926 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:52.994889    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:53.000327 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.000300    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:53.298195 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.298150    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:53.304050 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.304022    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:53.357410 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.357378    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:53.357888 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.357433    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:53.357888 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.357497    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:53.358100 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.358001    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:55:53.358157 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.358142    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:55:53.358881 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:53.358864    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:55:57.614627 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.614594    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"]
Oct 30 00:55:57.628443 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.628416    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:57.637855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.637818    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"]
Oct 30 00:55:57.646714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.646684    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c104b8\""
Oct 30 00:55:57.683280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.683235    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"]
Oct 30 00:55:57.687336 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.687310    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:57.702537 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.702513    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c104b8-dockercfg-sqgd5\""
Oct 30 00:55:57.704924 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.704886    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"]
Oct 30 00:55:57.778481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.774098    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801aa334-19ec-4ae7-9c34-19bab145c954-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:57.778481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.774201    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxzpr\" (UniqueName: \"kubernetes.io/projected/801aa334-19ec-4ae7-9c34-19bab145c954-kube-api-access-pxzpr\") pod \"payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:57.778481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.774232    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbf2e51-6a9c-4902-8438-ae95f266a431-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:57.778481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.774383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrzn\" (UniqueName: \"kubernetes.io/projected/abbf2e51-6a9c-4902-8438-ae95f266a431-kube-api-access-dkrzn\") pod \"payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:57.875525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.875415    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801aa334-19ec-4ae7-9c34-19bab145c954-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:57.875719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.875526    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxzpr\" (UniqueName: \"kubernetes.io/projected/801aa334-19ec-4ae7-9c34-19bab145c954-kube-api-access-pxzpr\") pod \"payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:57.875719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.875562    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbf2e51-6a9c-4902-8438-ae95f266a431-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:57.875719 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.875655    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrzn\" (UniqueName: \"kubernetes.io/projected/abbf2e51-6a9c-4902-8438-ae95f266a431-kube-api-access-dkrzn\") pod \"payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:57.876329 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.876306    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801aa334-19ec-4ae7-9c34-19bab145c954-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:57.876560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.876540    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbf2e51-6a9c-4902-8438-ae95f266a431-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:57.893818 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.893787    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrzn\" (UniqueName: \"kubernetes.io/projected/abbf2e51-6a9c-4902-8438-ae95f266a431-kube-api-access-dkrzn\") pod \"payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:57.893954 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.893832    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxzpr\" (UniqueName: \"kubernetes.io/projected/801aa334-19ec-4ae7-9c34-19bab145c954-kube-api-access-pxzpr\") pod \"payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:57.943570 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:57.943535    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:58.001706 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.001682    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:58.240843 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.240807    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"]
Oct 30 00:55:58.242862 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:58.242835    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbf2e51_6a9c_4902_8438_ae95f266a431.slice/crio-a7a9c7928af25aab67a5ecb4b548383f1027d5ba090e889f6a6dce8a4546b1ca WatchSource:0}: Error finding container a7a9c7928af25aab67a5ecb4b548383f1027d5ba090e889f6a6dce8a4546b1ca: Status 404 returned error can't find the container with id a7a9c7928af25aab67a5ecb4b548383f1027d5ba090e889f6a6dce8a4546b1ca
Oct 30 00:55:58.365996 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.365962    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"]
Oct 30 00:55:58.366877 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:58.366845    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801aa334_19ec_4ae7_9c34_19bab145c954.slice/crio-406e409a4ca6736d239d06384da4de126da4545a980323b574d10c4ee530bdfc WatchSource:0}: Error finding container 406e409a4ca6736d239d06384da4de126da4545a980323b574d10c4ee530bdfc: Status 404 returned error can't find the container with id 406e409a4ca6736d239d06384da4de126da4545a980323b574d10c4ee530bdfc
Oct 30 00:55:58.386671 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.386640    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" event={"ID":"801aa334-19ec-4ae7-9c34-19bab145c954","Type":"ContainerStarted","Data":"406e409a4ca6736d239d06384da4de126da4545a980323b574d10c4ee530bdfc"}
Oct 30 00:55:58.388318 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.388291    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" event={"ID":"abbf2e51-6a9c-4902-8438-ae95f266a431","Type":"ContainerStarted","Data":"db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302"}
Oct 30 00:55:58.388403 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.388336    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" event={"ID":"abbf2e51-6a9c-4902-8438-ae95f266a431","Type":"ContainerStarted","Data":"a7a9c7928af25aab67a5ecb4b548383f1027d5ba090e889f6a6dce8a4546b1ca"}
Oct 30 00:55:58.388544 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.388526    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:55:58.426646 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.426596    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" podStartSLOduration=1.426579979 podStartE2EDuration="1.426579979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:58.424870055 +0000 UTC m=+799.972560552" watchObservedRunningTime="2026-10-30 00:55:58.426579979 +0000 UTC m=+799.974270476"
Oct 30 00:55:58.451620 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.451542    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"]
Oct 30 00:55:58.455541 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.455520    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.471789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.471763    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ba3d7b\""
Oct 30 00:55:58.474292 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.474264    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"]
Oct 30 00:55:58.520798 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.520718    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"]
Oct 30 00:55:58.524601 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.524577    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:58.541093 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.541068    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ba3d7b-dockercfg-xpgvw\""
Oct 30 00:55:58.544500 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.544470    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"]
Oct 30 00:55:58.609736 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.609662    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69499e2a-f2b6-4d79-8132-ed3b23a62752-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.610226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.610197    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8c7\" (UniqueName: \"kubernetes.io/projected/69499e2a-f2b6-4d79-8132-ed3b23a62752-kube-api-access-hq8c7\") pod \"payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.711383 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.711342    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8857\" (UniqueName: \"kubernetes.io/projected/439a5722-bfae-47e0-b389-c682c528531a-kube-api-access-n8857\") pod \"payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:58.711787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.711394    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a5722-bfae-47e0-b389-c682c528531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:58.711787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.711612    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69499e2a-f2b6-4d79-8132-ed3b23a62752-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.711787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.711695    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8c7\" (UniqueName: \"kubernetes.io/projected/69499e2a-f2b6-4d79-8132-ed3b23a62752-kube-api-access-hq8c7\") pod \"payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.712450 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.712426    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69499e2a-f2b6-4d79-8132-ed3b23a62752-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.734128 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.734099    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8c7\" (UniqueName: \"kubernetes.io/projected/69499e2a-f2b6-4d79-8132-ed3b23a62752-kube-api-access-hq8c7\") pod \"payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.769840 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.769806    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:58.813379 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.813338    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a5722-bfae-47e0-b389-c682c528531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:58.813584 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.813541    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8857\" (UniqueName: \"kubernetes.io/projected/439a5722-bfae-47e0-b389-c682c528531a-kube-api-access-n8857\") pod \"payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:58.814739 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.814683    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a5722-bfae-47e0-b389-c682c528531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:58.834177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.833987    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8857\" (UniqueName: \"kubernetes.io/projected/439a5722-bfae-47e0-b389-c682c528531a-kube-api-access-n8857\") pod \"payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:58.843312 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:58.841823    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:59.086043 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.085012    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"]
Oct 30 00:55:59.089233 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:59.088323    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69499e2a_f2b6_4d79_8132_ed3b23a62752.slice/crio-297c6a3179514b11993a9675b8ac1fe4132797a9e2e5fb43c1f5ea6bc4dd9970 WatchSource:0}: Error finding container 297c6a3179514b11993a9675b8ac1fe4132797a9e2e5fb43c1f5ea6bc4dd9970: Status 404 returned error can't find the container with id 297c6a3179514b11993a9675b8ac1fe4132797a9e2e5fb43c1f5ea6bc4dd9970
Oct 30 00:55:59.151676 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.151632    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"]
Oct 30 00:55:59.154729 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:55:59.154699    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439a5722_bfae_47e0_b389_c682c528531a.slice/crio-1e97d04f40b76de5af77148e0bef0db0ad50d839479dc8d77e331e1b76b619b1 WatchSource:0}: Error finding container 1e97d04f40b76de5af77148e0bef0db0ad50d839479dc8d77e331e1b76b619b1: Status 404 returned error can't find the container with id 1e97d04f40b76de5af77148e0bef0db0ad50d839479dc8d77e331e1b76b619b1
Oct 30 00:55:59.396015 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.395914    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" event={"ID":"801aa334-19ec-4ae7-9c34-19bab145c954","Type":"ContainerStarted","Data":"2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b"}
Oct 30 00:55:59.396191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.396118    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:55:59.397698 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.397672    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" event={"ID":"439a5722-bfae-47e0-b389-c682c528531a","Type":"ContainerStarted","Data":"6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6"}
Oct 30 00:55:59.397841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.397714    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" event={"ID":"439a5722-bfae-47e0-b389-c682c528531a","Type":"ContainerStarted","Data":"1e97d04f40b76de5af77148e0bef0db0ad50d839479dc8d77e331e1b76b619b1"}
Oct 30 00:55:59.397841 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.397741    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:55:59.399536 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.399508    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" event={"ID":"69499e2a-f2b6-4d79-8132-ed3b23a62752","Type":"ContainerStarted","Data":"7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a"}
Oct 30 00:55:59.399681 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.399564    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" event={"ID":"69499e2a-f2b6-4d79-8132-ed3b23a62752","Type":"ContainerStarted","Data":"297c6a3179514b11993a9675b8ac1fe4132797a9e2e5fb43c1f5ea6bc4dd9970"}
Oct 30 00:55:59.399681 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.399601    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:55:59.433086 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.433028    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" podStartSLOduration=2.4330101 podStartE2EDuration="2.4330101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:59.431129989 +0000 UTC m=+800.978820486" watchObservedRunningTime="2026-10-30 00:55:59.4330101 +0000 UTC m=+800.980700599"
Oct 30 00:55:59.475791 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.475739    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" podStartSLOduration=1.475724011 podStartE2EDuration="1.475724011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:59.473847946 +0000 UTC m=+801.021538440" watchObservedRunningTime="2026-10-30 00:55:59.475724011 +0000 UTC m=+801.023414507"
Oct 30 00:55:59.510812 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:55:59.510753    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" podStartSLOduration=1.5107373210000001 podStartE2EDuration="1.510737321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:59.508440424 +0000 UTC m=+801.056130920" watchObservedRunningTime="2026-10-30 00:55:59.510737321 +0000 UTC m=+801.058427816"
Oct 30 00:56:00.079361 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.079322    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"]
Oct 30 00:56:00.083486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.083445    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.100819 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.100783    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"]
Oct 30 00:56:00.109394 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.109362    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-82b647-dockercfg-sgp29\""
Oct 30 00:56:00.109561 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.109362    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-82b647\""
Oct 30 00:56:00.198023 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.196136    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2hc\" (UniqueName: \"kubernetes.io/projected/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-kube-api-access-rx2hc\") pod \"payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.198023 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.196400    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.201585 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.201549    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"]
Oct 30 00:56:00.207349 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.207320    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.208977 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.208951    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"]
Oct 30 00:56:00.301418 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.301378    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.301649 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.301438    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.301649 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.301587    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2hc\" (UniqueName: \"kubernetes.io/projected/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-kube-api-access-rx2hc\") pod \"payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.301756 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.301651    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvfg\" (UniqueName: \"kubernetes.io/projected/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-kube-api-access-7dvfg\") pod \"payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.302229 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.302211    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.320354 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.320314    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2hc\" (UniqueName: \"kubernetes.io/projected/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-kube-api-access-rx2hc\") pod \"payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.402764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.402665    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:00.405366 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.403103    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvfg\" (UniqueName: \"kubernetes.io/projected/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-kube-api-access-7dvfg\") pod \"payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.405366 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.403253    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.405366 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.404755    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.423560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.423531    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvfg\" (UniqueName: \"kubernetes.io/projected/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-kube-api-access-7dvfg\") pod \"payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.531848 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.531816    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:00.703637 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.703112    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"]
Oct 30 00:56:00.706351 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:00.706254    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7427cc_2a6f_4b80_abaa_f41f3be1fea1.slice/crio-6ed6a9fcead33b3979adbb98fa4dadcb95f7d1fa509e3fbcc815893cc156e59e WatchSource:0}: Error finding container 6ed6a9fcead33b3979adbb98fa4dadcb95f7d1fa509e3fbcc815893cc156e59e: Status 404 returned error can't find the container with id 6ed6a9fcead33b3979adbb98fa4dadcb95f7d1fa509e3fbcc815893cc156e59e
Oct 30 00:56:00.840319 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:00.840293    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"]
Oct 30 00:56:00.842392 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:00.842354    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6c1f0a_bd07_4b25_ab2b_278b8d6c47d0.slice/crio-93e727a74588e9db427bbf6c97405f7ff6c560f201b8848ff20c8dd1d9b3b61e WatchSource:0}: Error finding container 93e727a74588e9db427bbf6c97405f7ff6c560f201b8848ff20c8dd1d9b3b61e: Status 404 returned error can't find the container with id 93e727a74588e9db427bbf6c97405f7ff6c560f201b8848ff20c8dd1d9b3b61e
Oct 30 00:56:01.156920 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.156806    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:56:01.419438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.419404    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" event={"ID":"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1","Type":"ContainerStarted","Data":"3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996"}
Oct 30 00:56:01.419644 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.419485    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:01.419644 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.419512    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" event={"ID":"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1","Type":"ContainerStarted","Data":"6ed6a9fcead33b3979adbb98fa4dadcb95f7d1fa509e3fbcc815893cc156e59e"}
Oct 30 00:56:01.421305 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.421281    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" event={"ID":"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0","Type":"ContainerStarted","Data":"d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311"}
Oct 30 00:56:01.421442 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.421328    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" event={"ID":"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0","Type":"ContainerStarted","Data":"93e727a74588e9db427bbf6c97405f7ff6c560f201b8848ff20c8dd1d9b3b61e"}
Oct 30 00:56:01.421442 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.421372    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:01.459517 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.459413    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" podStartSLOduration=1.45939325 podStartE2EDuration="1.45939325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:01.458332985 +0000 UTC m=+803.006023485" watchObservedRunningTime="2026-10-30 00:56:01.45939325 +0000 UTC m=+803.007083748"
Oct 30 00:56:01.500336 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:01.500277    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" podStartSLOduration=1.500259415 podStartE2EDuration="1.500259415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:01.497112922 +0000 UTC m=+803.044803419" watchObservedRunningTime="2026-10-30 00:56:01.500259415 +0000 UTC m=+803.047949911"
Oct 30 00:56:03.379622 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.379575    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"]
Oct 30 00:56:03.383753 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.383732    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:03.400619 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.400583    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"]
Oct 30 00:56:03.473478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.473251    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnjc\" (UniqueName: \"kubernetes.io/projected/4e2470c5-091d-4c27-86b4-f096085566ca-kube-api-access-kdnjc\") pod \"payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:03.473478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.473356    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2470c5-091d-4c27-86b4-f096085566ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:03.477481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.473850    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-c4847c\""
Oct 30 00:56:03.477481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.474006    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-c4847c-dockercfg-5qdcl\""
Oct 30 00:56:03.574561 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.574518    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnjc\" (UniqueName: \"kubernetes.io/projected/4e2470c5-091d-4c27-86b4-f096085566ca-kube-api-access-kdnjc\") pod \"payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:03.574750 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.574579    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2470c5-091d-4c27-86b4-f096085566ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:03.575449 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.575427    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2470c5-091d-4c27-86b4-f096085566ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:03.592726 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.592685    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnjc\" (UniqueName: \"kubernetes.io/projected/4e2470c5-091d-4c27-86b4-f096085566ca-kube-api-access-kdnjc\") pod \"payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:03.698106 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:03.698071    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:04.019746 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.018348    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"]
Oct 30 00:56:04.028499 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:04.021007    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2470c5_091d_4c27_86b4_f096085566ca.slice/crio-7c06e86916bf4473f50af603fe9c1a4787a12dfa8ae2ae950af9a20e58ffa261 WatchSource:0}: Error finding container 7c06e86916bf4473f50af603fe9c1a4787a12dfa8ae2ae950af9a20e58ffa261: Status 404 returned error can't find the container with id 7c06e86916bf4473f50af603fe9c1a4787a12dfa8ae2ae950af9a20e58ffa261
Oct 30 00:56:04.177290 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.177253    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"]
Oct 30 00:56:04.182199 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.182175    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.195677 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.195651    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d827b7\""
Oct 30 00:56:04.198281 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.198254    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"]
Oct 30 00:56:04.244823 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.244793    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"]
Oct 30 00:56:04.248859 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.248836    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.263577 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.263550    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-d827b7-dockercfg-bx7g2\""
Oct 30 00:56:04.266047 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.266021    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"]
Oct 30 00:56:04.282914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.282831    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.282914 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.282885    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxxx\" (UniqueName: \"kubernetes.io/projected/7aab52d9-16c4-470f-97ac-6adbf20090ee-kube-api-access-8lxxx\") pod \"payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.283087 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.282974    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aab52d9-16c4-470f-97ac-6adbf20090ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.283087 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.283051    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zgq\" (UniqueName: \"kubernetes.io/projected/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-kube-api-access-24zgq\") pod \"payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.386714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.384739    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.386714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.384819    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxxx\" (UniqueName: \"kubernetes.io/projected/7aab52d9-16c4-470f-97ac-6adbf20090ee-kube-api-access-8lxxx\") pod \"payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.386714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.384876    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aab52d9-16c4-470f-97ac-6adbf20090ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.386714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.384924    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zgq\" (UniqueName: \"kubernetes.io/projected/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-kube-api-access-24zgq\") pod \"payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.386714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.386205    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.387427 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.387318    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aab52d9-16c4-470f-97ac-6adbf20090ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.403838 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.403794    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zgq\" (UniqueName: \"kubernetes.io/projected/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-kube-api-access-24zgq\") pod \"payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.404655 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.404635    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxxx\" (UniqueName: \"kubernetes.io/projected/7aab52d9-16c4-470f-97ac-6adbf20090ee-kube-api-access-8lxxx\") pod \"payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.497569 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.497536    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:04.521026 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.520987    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" event={"ID":"4e2470c5-091d-4c27-86b4-f096085566ca","Type":"ContainerStarted","Data":"924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632"}
Oct 30 00:56:04.521187 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.521054    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" event={"ID":"4e2470c5-091d-4c27-86b4-f096085566ca","Type":"ContainerStarted","Data":"7c06e86916bf4473f50af603fe9c1a4787a12dfa8ae2ae950af9a20e58ffa261"}
Oct 30 00:56:04.521268 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.521189    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:04.559952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.559898    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" podStartSLOduration=1.5598801230000001 podStartE2EDuration="1.559880123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:04.556246464 +0000 UTC m=+806.103936961" watchObservedRunningTime="2026-10-30 00:56:04.559880123 +0000 UTC m=+806.107570616"
Oct 30 00:56:04.574714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.574681    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:04.799529 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.799497    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"]
Oct 30 00:56:04.802201 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:04.802169    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc18e1d_ab8c_411a_b514_2ab57ab74afa.slice/crio-3af1971a0b885b3dec3ec4b873cb6fb27a0674f38d1e6293d71c9ce27d94f68b WatchSource:0}: Error finding container 3af1971a0b885b3dec3ec4b873cb6fb27a0674f38d1e6293d71c9ce27d94f68b: Status 404 returned error can't find the container with id 3af1971a0b885b3dec3ec4b873cb6fb27a0674f38d1e6293d71c9ce27d94f68b
Oct 30 00:56:04.901495 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:04.901443    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"]
Oct 30 00:56:04.901942 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:04.901904    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aab52d9_16c4_470f_97ac_6adbf20090ee.slice/crio-bf57876ccc75d8c10c2637b1ccac1742bac9f1914c2a684dfdadeb9a397fe055 WatchSource:0}: Error finding container bf57876ccc75d8c10c2637b1ccac1742bac9f1914c2a684dfdadeb9a397fe055: Status 404 returned error can't find the container with id bf57876ccc75d8c10c2637b1ccac1742bac9f1914c2a684dfdadeb9a397fe055
Oct 30 00:56:05.528959 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.528918    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" event={"ID":"7aab52d9-16c4-470f-97ac-6adbf20090ee","Type":"ContainerStarted","Data":"59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c"}
Oct 30 00:56:05.529477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.528985    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" event={"ID":"7aab52d9-16c4-470f-97ac-6adbf20090ee","Type":"ContainerStarted","Data":"bf57876ccc75d8c10c2637b1ccac1742bac9f1914c2a684dfdadeb9a397fe055"}
Oct 30 00:56:05.529477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.529014    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:05.530822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.530797    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" event={"ID":"7cc18e1d-ab8c-411a-b514-2ab57ab74afa","Type":"ContainerStarted","Data":"be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac"}
Oct 30 00:56:05.530929 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.530846    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" event={"ID":"7cc18e1d-ab8c-411a-b514-2ab57ab74afa","Type":"ContainerStarted","Data":"3af1971a0b885b3dec3ec4b873cb6fb27a0674f38d1e6293d71c9ce27d94f68b"}
Oct 30 00:56:05.531045 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.531024    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:05.572796 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.572741    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" podStartSLOduration=1.572721904 podStartE2EDuration="1.572721904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:05.57067771 +0000 UTC m=+807.118368206" watchObservedRunningTime="2026-10-30 00:56:05.572721904 +0000 UTC m=+807.120412402"
Oct 30 00:56:05.609331 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:05.609267    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" podStartSLOduration=1.6092473680000001 podStartE2EDuration="1.609247368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:05.607746688 +0000 UTC m=+807.155437184" watchObservedRunningTime="2026-10-30 00:56:05.609247368 +0000 UTC m=+807.156937865"
Oct 30 00:56:09.118265 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:56:09.118227    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:56:09.404051 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:09.403962    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 00:56:10.414995 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:10.414961    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 00:56:10.415599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:10.415571    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 00:56:10.415798 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:10.415782    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 00:56:11.232857 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:11.232820    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:56:12.431549 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:12.431515    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 00:56:12.432382 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:12.432357    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 00:56:15.534233 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:15.534206    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 00:56:16.568482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:16.566666    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 00:56:16.570543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:16.569429    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 00:56:18.666180 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:18.666091    5104 generic.go:356] "Generic (PLEG): container finished" podID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerID="264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a" exitCode=0
Oct 30 00:56:18.666718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:18.666214    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" event={"ID":"17386f66-3f21-4f9e-8a12-9700848daaf9","Type":"ContainerDied","Data":"264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a"}
Oct 30 00:56:18.666718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:18.666677    5104 scope.go:122] "RemoveContainer" containerID="264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a"
Oct 30 00:56:18.667997 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:18.667966    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:18.668081 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:18.668062    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:18.668142 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:18.668097    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:19.696482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:19.693892    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" event={"ID":"17386f66-3f21-4f9e-8a12-9700848daaf9","Type":"ContainerStarted","Data":"f0ee5e068d57df3ddc8a7ca3b450467d9c14babfa329df065c99e1b2584a40d5"}
Oct 30 00:56:20.012844 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:20.010705    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:56:20.013195 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:20.012109    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:56:20.013792 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:20.013358    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 00:56:20.997477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:20.996935    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:56:20.997477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:20.996996    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 00:56:21.998369 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:21.997865    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:56:21.998369 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:21.997935    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 00:56:22.996483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:22.995971    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:56:22.996483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:22.996064    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 00:56:23.365542 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:23.364678    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:56:24.000482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:23.999923    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:56:24.000482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.000004    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 00:56:24.231477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.230929    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:56:24.415566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.412545    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:56:24.415566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.412612    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:24.720593 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.720549    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"]
Oct 30 00:56:24.726547 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.726520    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.759552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.755549    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"kube-root-ca.crt\""
Oct 30 00:56:24.759552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.755653    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"openshift-service-ca.crt\""
Oct 30 00:56:24.759552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.756543    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"default-dockercfg-c5kjw\""
Oct 30 00:56:24.759552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.756716    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"simulator-w5-dca38d-kserve-self-signed-certs\""
Oct 30 00:56:24.761746 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.761679    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"]
Oct 30 00:56:24.817089 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.817027    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-home\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.817398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.817361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-model-cache\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.817551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.817487    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-dshm\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.817551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.817520    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c35be6b-f3d3-4394-9e50-ff646a452e86-tls-certs\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.817771 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.817749    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzpvn\" (UniqueName: \"kubernetes.io/projected/3c35be6b-f3d3-4394-9e50-ff646a452e86-kube-api-access-lzpvn\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.817855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.817840    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-tmp-dir\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919025 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.918980    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-dshm\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919045    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c35be6b-f3d3-4394-9e50-ff646a452e86-tls-certs\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919102    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzpvn\" (UniqueName: \"kubernetes.io/projected/3c35be6b-f3d3-4394-9e50-ff646a452e86-kube-api-access-lzpvn\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919128    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-tmp-dir\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919174    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-home\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919345 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919210    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-model-cache\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919618 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919591    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-model-cache\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.919894 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919871    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-tmp-dir\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.920007 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.919987    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-home\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.922232 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.922185    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-dshm\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.923239 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.923193    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c35be6b-f3d3-4394-9e50-ff646a452e86-tls-certs\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:24.959418 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:24.959348    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzpvn\" (UniqueName: \"kubernetes.io/projected/3c35be6b-f3d3-4394-9e50-ff646a452e86-kube-api-access-lzpvn\") pod \"simulator-w5-dca38d-kserve-5b878fd8cf-tprg7\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") " pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:25.042704 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:25.042215    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:56:25.042704 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:25.042274    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 00:56:25.062171 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:25.061706    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:25.427393 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:25.427366    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"]
Oct 30 00:56:25.431080 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:25.431051    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c35be6b_f3d3_4394_9e50_ff646a452e86.slice/crio-6a3f47d175ad5f141f06bf1bf3e8e0b59d6894d5d816c966f623dd978066ed34 WatchSource:0}: Error finding container 6a3f47d175ad5f141f06bf1bf3e8e0b59d6894d5d816c966f623dd978066ed34: Status 404 returned error can't find the container with id 6a3f47d175ad5f141f06bf1bf3e8e0b59d6894d5d816c966f623dd978066ed34
Oct 30 00:56:25.890492 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:25.889784    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" event={"ID":"3c35be6b-f3d3-4394-9e50-ff646a452e86","Type":"ContainerStarted","Data":"364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577"}
Oct 30 00:56:25.890492 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:25.889856    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" event={"ID":"3c35be6b-f3d3-4394-9e50-ff646a452e86","Type":"ContainerStarted","Data":"6a3f47d175ad5f141f06bf1bf3e8e0b59d6894d5d816c966f623dd978066ed34"}
Oct 30 00:56:25.934565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:25.930449    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podStartSLOduration=1.930428853 podStartE2EDuration="1.930428853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:25.929766021 +0000 UTC m=+827.477456517" watchObservedRunningTime="2026-10-30 00:56:25.930428853 +0000 UTC m=+827.478119347"
Oct 30 00:56:26.073741 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:26.073635    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:26.254159 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:26.252779    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:56:26.270251 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:26.266552    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 00:56:28.914492 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.914389    5104 generic.go:356] "Generic (PLEG): container finished" podID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerID="5e1d3247a02de3ab22ef77909c2b6b5258322e042ad4492576e5842b4f139477" exitCode=0
Oct 30 00:56:28.915002 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.914480    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" event={"ID":"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a","Type":"ContainerDied","Data":"5e1d3247a02de3ab22ef77909c2b6b5258322e042ad4492576e5842b4f139477"}
Oct 30 00:56:28.915069 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.915007    5104 scope.go:122] "RemoveContainer" containerID="5e1d3247a02de3ab22ef77909c2b6b5258322e042ad4492576e5842b4f139477"
Oct 30 00:56:28.916514 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.916491    5104 generic.go:356] "Generic (PLEG): container finished" podID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerID="0a0bd83d509ee3c6cf4f080b5691f1eb6f9be5bdf00bbf00fd59b88ec95eaf25" exitCode=0
Oct 30 00:56:28.916614 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.916565    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" event={"ID":"d03188ad-e1df-4bdf-bcac-b9d46a5e704b","Type":"ContainerDied","Data":"0a0bd83d509ee3c6cf4f080b5691f1eb6f9be5bdf00bbf00fd59b88ec95eaf25"}
Oct 30 00:56:28.916614 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.916593    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:28.916696 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.916674    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:28.916742 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.916725    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:28.916992 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.916976    5104 scope.go:122] "RemoveContainer" containerID="0a0bd83d509ee3c6cf4f080b5691f1eb6f9be5bdf00bbf00fd59b88ec95eaf25"
Oct 30 00:56:28.918285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.918259    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:28.918364 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.918322    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:28.918405 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:28.918361    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:29.938750 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.938707    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" event={"ID":"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a","Type":"ContainerStarted","Data":"5e5f0a671a4f67b6aee40117e7055b4dc70d95bcf749e4943345115897ef36e7"}
Oct 30 00:56:29.942356 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.942329    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" event={"ID":"d03188ad-e1df-4bdf-bcac-b9d46a5e704b","Type":"ContainerStarted","Data":"2cafa7121ee5459132b463806bad6c066b4eb80ee3e6051e10422fb0e9c47a8f"}
Oct 30 00:56:29.944324 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.944290    5104 generic.go:356] "Generic (PLEG): container finished" podID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerID="c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a" exitCode=0
Oct 30 00:56:29.944503 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.944372    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" event={"ID":"65623efa-d0c0-455c-a784-3b48e1e9eb24","Type":"ContainerDied","Data":"c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a"}
Oct 30 00:56:29.944737 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.944714    5104 scope.go:122] "RemoveContainer" containerID="c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a"
Oct 30 00:56:29.946165 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.946129    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:29.946271 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.946197    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:29.946271 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:29.946226    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:56:30.334479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:30.315810    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:56:30.334828 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:30.317584    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:56:30.349605 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:30.349565    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 00:56:30.852477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:30.851949    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:56:30.852686 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:30.852530    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:30.852686 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:30.852587    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 00:56:30.953082 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:30.953047    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" event={"ID":"65623efa-d0c0-455c-a784-3b48e1e9eb24","Type":"ContainerStarted","Data":"37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa"}
Oct 30 00:56:31.305495 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:31.297908    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:56:31.305495 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:31.297975    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 00:56:31.380483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:31.379882    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:56:31.380483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:31.380016    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:56:31.380483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:31.380062    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:56:31.853480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:31.852932    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:31.853480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:31.852998    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 00:56:32.299564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:32.299530    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:56:32.300173 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:32.300139    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 00:56:32.380761 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:32.380722    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:56:32.381055 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:32.381020    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:56:32.852479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:32.851874    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:32.852479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:32.851942    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 00:56:33.299926 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:33.299889    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:56:33.300590 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:33.300548    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 00:56:33.377059 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:33.377020    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:56:33.377385 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:33.377342    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:56:33.950561 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:33.949602    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:33.950561 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:33.949668    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 00:56:34.299542 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:34.297894    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:56:34.299542 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:34.297964    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 00:56:34.379951 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:34.379906    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:56:34.380597 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:34.380563    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:56:34.850484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:34.849788    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:34.850484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:34.849846    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 00:56:35.063549 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.063518    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:35.113922 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.113830    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:35.300925 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.300885    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:56:35.301201 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.301172    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 00:56:35.377025 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.376931    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:56:35.377304 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.377274    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:56:35.871859 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.871819    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:56:35.876546 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.876509    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:56:35.913378 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:35.909999    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 00:56:36.061656 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:36.061620    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:56:36.394481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:36.392861    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 00:56:36.394481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:36.392925    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 00:56:36.437482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:36.433951    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:56:36.437482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:36.435352    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:56:36.437482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:36.436658    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 00:56:37.006535 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.006493    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:56:37.352280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.352183    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:56:37.436467 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.436430    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"]
Oct 30 00:56:37.443328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.442175    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.467980 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.467938    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"kube-root-ca.crt\""
Oct 30 00:56:37.468132 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.468040    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"openshift-service-ca.crt\""
Oct 30 00:56:37.472517 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.472491    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"]
Oct 30 00:56:37.484815 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.484770    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"default-dockercfg-tmv96\""
Oct 30 00:56:37.484815 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.484791    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"simulator-w4-d827b7-kserve-self-signed-certs\""
Oct 30 00:56:37.523721 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.523671    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:56:37.558498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.558444    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d558cb04-b45c-478a-aa73-b0862f1d98c9-tls-certs\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.558685 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.558516    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjd7l\" (UniqueName: \"kubernetes.io/projected/d558cb04-b45c-478a-aa73-b0862f1d98c9-kube-api-access-jjd7l\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.558685 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.558670    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-model-cache\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.558798 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.558714    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-home\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.558798 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.558771    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-tmp-dir\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.558798 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.558796    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-dshm\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.659773 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.659683    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-model-cache\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.659773 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.659741    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-home\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.659773 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.659770    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-tmp-dir\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.660070 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.659795    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-dshm\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.660070 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.659850    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d558cb04-b45c-478a-aa73-b0862f1d98c9-tls-certs\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.660070 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.659887    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjd7l\" (UniqueName: \"kubernetes.io/projected/d558cb04-b45c-478a-aa73-b0862f1d98c9-kube-api-access-jjd7l\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.660227 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.660175    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-model-cache\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.660227 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.660182    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-tmp-dir\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.660306 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.660239    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-home\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.662243 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.662216    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-dshm\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.662529 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.662504    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d558cb04-b45c-478a-aa73-b0862f1d98c9-tls-certs\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.683509 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.683447    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjd7l\" (UniqueName: \"kubernetes.io/projected/d558cb04-b45c-478a-aa73-b0862f1d98c9-kube-api-access-jjd7l\") pod \"simulator-w4-d827b7-kserve-844876ff99-2fdw9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") " pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:37.762652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:37.762331    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:38.063996 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:38.063959    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 00:56:38.107804 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:38.107773    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"]
Oct 30 00:56:38.110885 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:38.110312    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd558cb04_b45c_478a_aa73_b0862f1d98c9.slice/crio-51d1fe0f2211819080a3a5e1e84783aef578bcc37f8016c391eabf76255408ed WatchSource:0}: Error finding container 51d1fe0f2211819080a3a5e1e84783aef578bcc37f8016c391eabf76255408ed: Status 404 returned error can't find the container with id 51d1fe0f2211819080a3a5e1e84783aef578bcc37f8016c391eabf76255408ed
Oct 30 00:56:39.069096 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:39.069057    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" event={"ID":"d558cb04-b45c-478a-aa73-b0862f1d98c9","Type":"ContainerStarted","Data":"bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a"}
Oct 30 00:56:39.069490 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:39.069107    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" event={"ID":"d558cb04-b45c-478a-aa73-b0862f1d98c9","Type":"ContainerStarted","Data":"51d1fe0f2211819080a3a5e1e84783aef578bcc37f8016c391eabf76255408ed"}
Oct 30 00:56:39.104152 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:56:39.104107    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:56:39.105573 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:39.105527    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podStartSLOduration=2.105509724 podStartE2EDuration="2.105509724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:39.105293439 +0000 UTC m=+840.652983937" watchObservedRunningTime="2026-10-30 00:56:39.105509724 +0000 UTC m=+840.653200244"
Oct 30 00:56:39.763002 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:39.762964    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:46.827966 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:46.827931    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:56:47.763636 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:47.763601    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:47.776171 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:47.776145    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:48.133351 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:48.133276    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:56:59.140264 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.140173    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"]
Oct 30 00:56:59.145162 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.145136    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.162106 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.162057    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"]
Oct 30 00:56:59.169634 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.169605    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"openshift-service-ca.crt\""
Oct 30 00:56:59.169789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.169723    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"kube-root-ca.crt\""
Oct 30 00:56:59.170358 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.170338    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"default-dockercfg-vh452\""
Oct 30 00:56:59.170358 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.170346    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"premium-w0-ba3d7b-kserve-self-signed-certs\""
Oct 30 00:56:59.246044 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.246003    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-home\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.246175 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.246126    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f44d4e-7208-498f-bced-9b20007068b9-tls-certs\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.246246 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.246231    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-tmp-dir\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.246351 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.246333    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-dshm\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.246425 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.246405    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-model-cache\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.246520 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.246442    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7gc\" (UniqueName: \"kubernetes.io/projected/85f44d4e-7208-498f-bced-9b20007068b9-kube-api-access-2z7gc\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.357275 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.357228    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f44d4e-7208-498f-bced-9b20007068b9-tls-certs\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.357668 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.357645    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-tmp-dir\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.357865 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.357849    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-dshm\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.357982 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.357969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-model-cache\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.358083 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.358069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7gc\" (UniqueName: \"kubernetes.io/projected/85f44d4e-7208-498f-bced-9b20007068b9-kube-api-access-2z7gc\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.358197 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.358184    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-home\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.358764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.358739    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-home\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.362046 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.360786    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-tmp-dir\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.362046 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.361040    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-model-cache\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.369269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.369237    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f44d4e-7208-498f-bced-9b20007068b9-tls-certs\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.369426 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.369401    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-dshm\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.383384 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.383349    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7gc\" (UniqueName: \"kubernetes.io/projected/85f44d4e-7208-498f-bced-9b20007068b9-kube-api-access-2z7gc\") pod \"premium-w0-ba3d7b-kserve-6598984495-gsnfc\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") " pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.487652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.487613    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:56:59.785056 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:56:59.785026    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"]
Oct 30 00:56:59.787623 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:56:59.787593    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85f44d4e_7208_498f_bced_9b20007068b9.slice/crio-5201bb9486c21f3271bc056f5c1dce9db2393bd3e51df6de91111fec4dcce313 WatchSource:0}: Error finding container 5201bb9486c21f3271bc056f5c1dce9db2393bd3e51df6de91111fec4dcce313: Status 404 returned error can't find the container with id 5201bb9486c21f3271bc056f5c1dce9db2393bd3e51df6de91111fec4dcce313
Oct 30 00:57:00.219133 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:00.219094    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" event={"ID":"85f44d4e-7208-498f-bced-9b20007068b9","Type":"ContainerStarted","Data":"f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95"}
Oct 30 00:57:00.219562 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:00.219141    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" event={"ID":"85f44d4e-7208-498f-bced-9b20007068b9","Type":"ContainerStarted","Data":"5201bb9486c21f3271bc056f5c1dce9db2393bd3e51df6de91111fec4dcce313"}
Oct 30 00:57:00.263359 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:00.263309    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podStartSLOduration=1.26329085 podStartE2EDuration="1.26329085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:00.26204753 +0000 UTC m=+861.809738026" watchObservedRunningTime="2026-10-30 00:57:00.26329085 +0000 UTC m=+861.810981346"
Oct 30 00:57:00.488598 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:00.488502    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:57:03.568429 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:03.568395    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:57:09.104907 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:57:09.104869    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:57:09.488733 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.488696    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:57:09.501832 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.501805    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:57:09.935226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.935190    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"]
Oct 30 00:57:09.941909 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.941877    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:09.959419 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.959201    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"]
Oct 30 00:57:09.966677 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.966649    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"openshift-service-ca.crt\""
Oct 30 00:57:09.966808 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.966698    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"default-dockercfg-w6qhr\""
Oct 30 00:57:09.966861 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.966813    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"premium-w6-c4847c-kserve-self-signed-certs\""
Oct 30 00:57:09.966950 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.966936    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"kube-root-ca.crt\""
Oct 30 00:57:09.985519 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.985485    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f21280-ac89-4809-9826-26ad3e76020b-tls-certs\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:09.985683 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.985539    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-home\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:09.985683 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.985585    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-tmp-dir\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:09.985683 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.985613    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-dshm\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:09.985791 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.985768    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr56r\" (UniqueName: \"kubernetes.io/projected/70f21280-ac89-4809-9826-26ad3e76020b-kube-api-access-tr56r\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:09.985825 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:09.985815    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-model-cache\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087024 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.086987    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f21280-ac89-4809-9826-26ad3e76020b-tls-certs\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087229 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087037    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-home\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087229 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087073    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-tmp-dir\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087229 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087098    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-dshm\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087396 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087243    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr56r\" (UniqueName: \"kubernetes.io/projected/70f21280-ac89-4809-9826-26ad3e76020b-kube-api-access-tr56r\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087396 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087283    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-model-cache\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087615 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087594    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-home\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087748 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087656    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-tmp-dir\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.087950 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.087923    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-model-cache\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.089560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.089536    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-dshm\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.089847 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.089826    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f21280-ac89-4809-9826-26ad3e76020b-tls-certs\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.136108 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.135064    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr56r\" (UniqueName: \"kubernetes.io/projected/70f21280-ac89-4809-9826-26ad3e76020b-kube-api-access-tr56r\") pod \"premium-w6-c4847c-kserve-64f7d94747-74fbh\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") " pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.260383 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.260285    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:10.299857 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.299824    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:57:10.555544 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:10.555516    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"]
Oct 30 00:57:10.557421 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:57:10.557388    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70f21280_ac89_4809_9826_26ad3e76020b.slice/crio-8eb274adfdf258ad9a6a0e74e5dc9f3368fa4702544ec2421aa1318fee912ea0 WatchSource:0}: Error finding container 8eb274adfdf258ad9a6a0e74e5dc9f3368fa4702544ec2421aa1318fee912ea0: Status 404 returned error can't find the container with id 8eb274adfdf258ad9a6a0e74e5dc9f3368fa4702544ec2421aa1318fee912ea0
Oct 30 00:57:11.295999 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:11.295965    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" event={"ID":"70f21280-ac89-4809-9826-26ad3e76020b","Type":"ContainerStarted","Data":"49e6d4f359d1d91023efd765e156e1e05a9a22128fa2e2b6faa412fbfe88e3ac"}
Oct 30 00:57:11.296430 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:11.296015    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" event={"ID":"70f21280-ac89-4809-9826-26ad3e76020b","Type":"ContainerStarted","Data":"8eb274adfdf258ad9a6a0e74e5dc9f3368fa4702544ec2421aa1318fee912ea0"}
Oct 30 00:57:11.332523 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:11.332465    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podStartSLOduration=2.332436925 podStartE2EDuration="2.332436925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:11.331320548 +0000 UTC m=+872.879011045" watchObservedRunningTime="2026-10-30 00:57:11.332436925 +0000 UTC m=+872.880127421"
Oct 30 00:57:12.260425 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:12.260390    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:20.260870 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:20.260832    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:20.279102 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:20.279073    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:20.372838 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:20.372805    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:57:25.123078 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.123038    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"]
Oct 30 00:57:25.127515 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.127492    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.144434 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.144404    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"]
Oct 30 00:57:25.152444 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.152413    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"openshift-service-ca.crt\""
Oct 30 00:57:25.152444 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.152478    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"kube-root-ca.crt\""
Oct 30 00:57:25.152709 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.152574    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"premium-w2-fd3281-kserve-self-signed-certs\""
Oct 30 00:57:25.152709 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.152679    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"default-dockercfg-5p7xd\""
Oct 30 00:57:25.207587 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.207535    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-model-cache\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.207759 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.207673    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckjr\" (UniqueName: \"kubernetes.io/projected/e69a955d-b56b-4dcd-943a-c1af4d8eb363-kube-api-access-7ckjr\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.207851 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.207768    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-home\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.207851 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.207831    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-dshm\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.207960 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.207930    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tmp-dir\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.208014 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.207959    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tls-certs\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.320476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.309290    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-model-cache\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.320476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.314551    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckjr\" (UniqueName: \"kubernetes.io/projected/e69a955d-b56b-4dcd-943a-c1af4d8eb363-kube-api-access-7ckjr\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.320476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.314692    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-home\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.320476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.314717    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-dshm\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.320476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.314797    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tmp-dir\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.320476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.314828    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tls-certs\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.342481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.321778    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-model-cache\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.342481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.335931    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tls-certs\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.342481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.336235    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-home\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.342481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.339603    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckjr\" (UniqueName: \"kubernetes.io/projected/e69a955d-b56b-4dcd-943a-c1af4d8eb363-kube-api-access-7ckjr\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.342481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.339876    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tmp-dir\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.363482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.346767    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-dshm\") pod \"premium-w2-fd3281-kserve-7ffcc4799b-58zkc\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") " pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.442803 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.442766    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:25.579899 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.578400    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:57:25.765272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:25.765247    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"]
Oct 30 00:57:25.767241 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:57:25.767213    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69a955d_b56b_4dcd_943a_c1af4d8eb363.slice/crio-b188667233dfe138234983122f0d72a9b2b9c368b0a57a4a7ff0bc71e825daf1 WatchSource:0}: Error finding container b188667233dfe138234983122f0d72a9b2b9c368b0a57a4a7ff0bc71e825daf1: Status 404 returned error can't find the container with id b188667233dfe138234983122f0d72a9b2b9c368b0a57a4a7ff0bc71e825daf1
Oct 30 00:57:26.404686 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:26.404650    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" event={"ID":"e69a955d-b56b-4dcd-943a-c1af4d8eb363","Type":"ContainerStarted","Data":"36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e"}
Oct 30 00:57:26.405084 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:26.404699    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" event={"ID":"e69a955d-b56b-4dcd-943a-c1af4d8eb363","Type":"ContainerStarted","Data":"b188667233dfe138234983122f0d72a9b2b9c368b0a57a4a7ff0bc71e825daf1"}
Oct 30 00:57:26.443444 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:26.443409    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:26.450651 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:26.450603    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podStartSLOduration=1.450588689 podStartE2EDuration="1.450588689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:26.448395856 +0000 UTC m=+887.996086352" watchObservedRunningTime="2026-10-30 00:57:26.450588689 +0000 UTC m=+887.998279185"
Oct 30 00:57:35.165384 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.165343    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"]
Oct 30 00:57:35.170551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.170533    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.187720 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.187684    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"]
Oct 30 00:57:35.195018 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.194993    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:35.195219 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.195080    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:57:35.195346 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.195236    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:57:35.199714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.198599    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:57:35.264219 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.264179    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.264431 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.264412    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.264552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.264503    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdhp\" (UniqueName: \"kubernetes.io/projected/4a052ef3-84a6-442f-be2d-096077d985d5-kube-api-access-csdhp\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.366057 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.366023    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.366240 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.366148    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.366240 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.366178    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdhp\" (UniqueName: \"kubernetes.io/projected/4a052ef3-84a6-442f-be2d-096077d985d5-kube-api-access-csdhp\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.366742 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.366717    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.368870 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.368839    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.386498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.386448    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdhp\" (UniqueName: \"kubernetes.io/projected/4a052ef3-84a6-442f-be2d-096077d985d5-kube-api-access-csdhp\") pod \"maas-api-revoke-sub-dd08eba6-hxzdj\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.443082 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.443049    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:35.456034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.456003    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:35.471820 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.471791    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:57:35.486101 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.486072    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:35.796606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:35.796449    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"]
Oct 30 00:57:35.798248 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:57:35.798211    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a052ef3_84a6_442f_be2d_096077d985d5.slice/crio-186886c453da743f965ae870d84c349bd84e92361ec9f4a7b943e97e86692833 WatchSource:0}: Error finding container 186886c453da743f965ae870d84c349bd84e92361ec9f4a7b943e97e86692833: Status 404 returned error can't find the container with id 186886c453da743f965ae870d84c349bd84e92361ec9f4a7b943e97e86692833
Oct 30 00:57:36.107227 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.107133    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"]
Oct 30 00:57:36.114443 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.114416    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.131809 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.131769    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"]
Oct 30 00:57:36.140842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.140332    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ba3d7b-openshift-default-dockercfg-6zwjk\""
Oct 30 00:57:36.278479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.278479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278414    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.279034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278505    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.279034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278594    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.279034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278621    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.279034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278651    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.279034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278704    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.279034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278756    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.279034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.278846    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7nn\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-kube-api-access-hv7nn\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381220    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381322    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7nn\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-kube-api-access-hv7nn\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381395    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381424    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381499    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381595    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381642    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381674    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.381725    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.382532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.382211    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.383155 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.382577    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.386478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.383550    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.396488 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.387934    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.396488 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.391186    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.400493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.399248    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.413834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.413782    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.417251 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.417218    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7nn\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-kube-api-access-hv7nn\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.433516 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.433039    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.433516 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.433276    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:36.473075 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.472548    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj" event={"ID":"4a052ef3-84a6-442f-be2d-096077d985d5","Type":"ContainerStarted","Data":"186886c453da743f965ae870d84c349bd84e92361ec9f4a7b943e97e86692833"}
Oct 30 00:57:36.787644 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.787612    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"]
Oct 30 00:57:36.794829 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:57:36.794792    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97970c7_2b74_4a0d_b5f6_698ebbe69bd8.slice/crio-08546feb24d59a6ae8928007b57abe0ecc5d7dafba64494ed0c6585c91d1717c WatchSource:0}: Error finding container 08546feb24d59a6ae8928007b57abe0ecc5d7dafba64494ed0c6585c91d1717c: Status 404 returned error can't find the container with id 08546feb24d59a6ae8928007b57abe0ecc5d7dafba64494ed0c6585c91d1717c
Oct 30 00:57:36.798053 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.798008    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:57:36.798174 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.798101    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:57:36.798174 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:36.798148    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:57:37.515206 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:37.515162    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" event={"ID":"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8","Type":"ContainerStarted","Data":"70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea"}
Oct 30 00:57:37.515749 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:37.515221    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" event={"ID":"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8","Type":"ContainerStarted","Data":"08546feb24d59a6ae8928007b57abe0ecc5d7dafba64494ed0c6585c91d1717c"}
Oct 30 00:57:37.586553 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:37.586487    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podStartSLOduration=1.586445544 podStartE2EDuration="1.586445544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:37.572130604 +0000 UTC m=+899.119821100" watchObservedRunningTime="2026-10-30 00:57:37.586445544 +0000 UTC m=+899.134136040"
Oct 30 00:57:38.116398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.113568    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"]
Oct 30 00:57:38.120951 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.120924    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"]
Oct 30 00:57:38.121287 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.121272    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.221572 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.221531    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.221769 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.221659    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.221832 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.221812    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcf8t\" (UniqueName: \"kubernetes.io/projected/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-kube-api-access-gcf8t\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.322913 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.322879    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcf8t\" (UniqueName: \"kubernetes.io/projected/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-kube-api-access-gcf8t\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.323123 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.323047    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.323203 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.323118    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.324037 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.324006    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.325652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.325623    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.340898 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.340866    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcf8t\" (UniqueName: \"kubernetes.io/projected/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-kube-api-access-gcf8t\") pod \"maas-api-revoke-sub-cc43d717-mldtb\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.434828 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.434795    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:38.436663 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.436633    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:57:38.436810 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.436683    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:57:38.442869 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.442844    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:38.667789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.667725    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:57:38.711403 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.711301    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 00:57:38.713925 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.713901    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:57:38.722222 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.722198    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 00:57:38.777500 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:38.773904    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"]
Oct 30 00:57:39.141359 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:57:39.141271    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:57:39.446904 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:39.438590    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:57:39.446904 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:39.438665    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:57:39.533594 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:39.533524    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb" event={"ID":"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490","Type":"ContainerStarted","Data":"65634ead385548e4de147984787c8f564cf6765a0b010e7d7be8392478247350"}
Oct 30 00:57:40.434699 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:40.434667    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:57:40.435123 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:40.434722    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:57:41.434872 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:41.434832    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:57:41.435366 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:41.434897    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:57:42.023417 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.023383    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"]
Oct 30 00:57:42.028639 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.028620    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.045345 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.045313    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"]
Oct 30 00:57:42.071551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.071436    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkb9p\" (UniqueName: \"kubernetes.io/projected/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-kube-api-access-vkb9p\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.071723 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.071565    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.071823 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.071805    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.176482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.172695    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkb9p\" (UniqueName: \"kubernetes.io/projected/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-kube-api-access-vkb9p\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.176482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.172755    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.176482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.172907    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.187986 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.186189    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.200949 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.200916    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.203177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.203152    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkb9p\" (UniqueName: \"kubernetes.io/projected/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-kube-api-access-vkb9p\") pod \"maas-api-revoke-sub-c3971a38-f57vn\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.343120 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.343033    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:42.434073 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.434040    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:57:42.434272 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.434098    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 00:57:42.563490 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.559548    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb" event={"ID":"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490","Type":"ContainerStarted","Data":"62f15140599819b6d744de89704b2c8093d7e17a17c8d0ac97a808dac9b5c09c"}
Oct 30 00:57:42.603535 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.602877    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb" podStartSLOduration=1.601886667 podStartE2EDuration="4.602858535s" totalImagesPullingTime="3.000971868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:42.602204942 +0000 UTC m=+904.149895440" watchObservedRunningTime="2026-10-30 00:57:42.602858535 +0000 UTC m=+904.150549032"
Oct 30 00:57:42.771057 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:42.771029    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"]
Oct 30 00:57:43.571594 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.571494    5104 generic.go:356] "Generic (PLEG): container finished" podID="f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" containerID="62f15140599819b6d744de89704b2c8093d7e17a17c8d0ac97a808dac9b5c09c" exitCode=0
Oct 30 00:57:43.572082 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.571627    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb" event={"ID":"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490","Type":"ContainerDied","Data":"62f15140599819b6d744de89704b2c8093d7e17a17c8d0ac97a808dac9b5c09c"}
Oct 30 00:57:43.573609 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.573584    5104 generic.go:356] "Generic (PLEG): container finished" podID="4a052ef3-84a6-442f-be2d-096077d985d5" containerID="99ec928716f330092d20c3cf1fe07c2d5c66b251629e3ec1c8bdd9d74b2f8e18" exitCode=0
Oct 30 00:57:43.573745 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.573672    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj" event={"ID":"4a052ef3-84a6-442f-be2d-096077d985d5","Type":"ContainerDied","Data":"99ec928716f330092d20c3cf1fe07c2d5c66b251629e3ec1c8bdd9d74b2f8e18"}
Oct 30 00:57:43.575638 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.575600    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn" event={"ID":"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23","Type":"ContainerStarted","Data":"8b26ee47b0c6aa48eeb3e70859a90ef8f89e22968cecf1d6d019a78e2548d227"}
Oct 30 00:57:43.575743 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.575662    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn" event={"ID":"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23","Type":"ContainerStarted","Data":"e14bb1f4420fb7a153218360955d7b7103983957c39218c836cda196b508ec7a"}
Oct 30 00:57:43.678134 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.678102    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:43.678792 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.678491    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:43.680587 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.680553    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 00:57:43.681277 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:43.681066    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn" podStartSLOduration=1.6810511780000001 podStartE2EDuration="1.681051178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:43.679016994 +0000 UTC m=+905.226707490" watchObservedRunningTime="2026-10-30 00:57:43.681051178 +0000 UTC m=+905.228741674"
Oct 30 00:57:44.582790 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.582756    5104 generic.go:356] "Generic (PLEG): container finished" podID="a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" containerID="8b26ee47b0c6aa48eeb3e70859a90ef8f89e22968cecf1d6d019a78e2548d227" exitCode=0
Oct 30 00:57:44.583262 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.582909    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn" event={"ID":"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23","Type":"ContainerDied","Data":"8b26ee47b0c6aa48eeb3e70859a90ef8f89e22968cecf1d6d019a78e2548d227"}
Oct 30 00:57:44.791293 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.790771    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:44.838191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.838107    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:44.935441 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.935409    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-xks-ca\") pod \"4a052ef3-84a6-442f-be2d-096077d985d5\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") "
Oct 30 00:57:44.935660 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.935483    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-xks-ca\") pod \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") "
Oct 30 00:57:44.935660 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.935595    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-kube-api-access-gcf8t\" (UniqueName: \"kubernetes.io/projected/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-kube-api-access-gcf8t\") pod \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") "
Oct 30 00:57:44.935760 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.935659    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-openshift-ca\") pod \"4a052ef3-84a6-442f-be2d-096077d985d5\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") "
Oct 30 00:57:44.935760 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.935739    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-openshift-ca\") pod \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\" (UID: \"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490\") "
Oct 30 00:57:44.935867 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.935797    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a052ef3-84a6-442f-be2d-096077d985d5-kube-api-access-csdhp\" (UniqueName: \"kubernetes.io/projected/4a052ef3-84a6-442f-be2d-096077d985d5-kube-api-access-csdhp\") pod \"4a052ef3-84a6-442f-be2d-096077d985d5\" (UID: \"4a052ef3-84a6-442f-be2d-096077d985d5\") "
Oct 30 00:57:44.936123 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.936090    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-openshift-ca" pod "4a052ef3-84a6-442f-be2d-096077d985d5" (UID: "4a052ef3-84a6-442f-be2d-096077d985d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:44.936233 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.936127    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-openshift-ca" pod "f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" (UID: "f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:44.936333 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.936314    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:44.936476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.936334    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:44.938005 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.937979    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-xks-ca" pod "f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" (UID: "f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:44.938205 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.938183    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-xks-ca" pod "4a052ef3-84a6-442f-be2d-096077d985d5" (UID: "4a052ef3-84a6-442f-be2d-096077d985d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:44.938265 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.938212    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-kube-api-access-gcf8t" pod "f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" (UID: "f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490"). InnerVolumeSpecName "kube-api-access-gcf8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:44.938406 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:44.938388    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a052ef3-84a6-442f-be2d-096077d985d5-kube-api-access-csdhp" pod "4a052ef3-84a6-442f-be2d-096077d985d5" (UID: "4a052ef3-84a6-442f-be2d-096077d985d5"). InnerVolumeSpecName "kube-api-access-csdhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:45.037089 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.037054    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdhp\" (UniqueName: \"kubernetes.io/projected/4a052ef3-84a6-442f-be2d-096077d985d5-kube-api-access-csdhp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:45.037089 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.037084    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a052ef3-84a6-442f-be2d-096077d985d5-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:45.037089 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.037093    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:45.037331 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.037102    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcf8t\" (UniqueName: \"kubernetes.io/projected/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490-kube-api-access-gcf8t\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:45.590340 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.590308    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"
Oct 30 00:57:45.590340 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.590319    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb" event={"ID":"f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490","Type":"ContainerDied","Data":"65634ead385548e4de147984787c8f564cf6765a0b010e7d7be8392478247350"}
Oct 30 00:57:45.590908 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.590371    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65634ead385548e4de147984787c8f564cf6765a0b010e7d7be8392478247350"
Oct 30 00:57:45.591811 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.591779    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj" event={"ID":"4a052ef3-84a6-442f-be2d-096077d985d5","Type":"ContainerDied","Data":"186886c453da743f965ae870d84c349bd84e92361ec9f4a7b943e97e86692833"}
Oct 30 00:57:45.591942 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.591826    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186886c453da743f965ae870d84c349bd84e92361ec9f4a7b943e97e86692833"
Oct 30 00:57:45.591942 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.591870    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"
Oct 30 00:57:45.722621 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.722590    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:45.870533 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.869798    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-openshift-ca\") pod \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") "
Oct 30 00:57:45.870533 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.869909    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-xks-ca\") pod \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") "
Oct 30 00:57:45.870533 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.870008    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-kube-api-access-vkb9p\" (UniqueName: \"kubernetes.io/projected/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-kube-api-access-vkb9p\") pod \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\" (UID: \"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23\") "
Oct 30 00:57:45.871517 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.871420    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-openshift-ca" pod "a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" (UID: "a6d16483-dc6b-4ff6-bc16-a6c0606e6b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:45.880654 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.880618    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-kube-api-access-vkb9p" pod "a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" (UID: "a6d16483-dc6b-4ff6-bc16-a6c0606e6b23"). InnerVolumeSpecName "kube-api-access-vkb9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:45.881856 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.881387    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-xks-ca" pod "a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" (UID: "a6d16483-dc6b-4ff6-bc16-a6c0606e6b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:45.972115 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.972083    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:45.972115 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.972114    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:45.972115 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:45.972124    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkb9p\" (UniqueName: \"kubernetes.io/projected/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23-kube-api-access-vkb9p\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:46.601254 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:46.601224    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"
Oct 30 00:57:46.601254 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:46.601228    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn" event={"ID":"a6d16483-dc6b-4ff6-bc16-a6c0606e6b23","Type":"ContainerDied","Data":"e14bb1f4420fb7a153218360955d7b7103983957c39218c836cda196b508ec7a"}
Oct 30 00:57:46.601666 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:46.601296    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14bb1f4420fb7a153218360955d7b7103983957c39218c836cda196b508ec7a"
Oct 30 00:57:50.711152 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:50.711117    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:57:52.516555 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.516515    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"]
Oct 30 00:57:52.517982 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.517954    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" containerName="revoke-keys"
Oct 30 00:57:52.518146 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.518130    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" containerName="revoke-keys"
Oct 30 00:57:52.518243 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.518231    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" containerName="revoke-keys"
Oct 30 00:57:52.518335 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.518324    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" containerName="revoke-keys"
Oct 30 00:57:52.518483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.518469    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4a052ef3-84a6-442f-be2d-096077d985d5" containerName="revoke-keys"
Oct 30 00:57:52.519796 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.519777    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a052ef3-84a6-442f-be2d-096077d985d5" containerName="revoke-keys"
Oct 30 00:57:52.523477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.520130    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a052ef3-84a6-442f-be2d-096077d985d5" containerName="revoke-keys"
Oct 30 00:57:52.523639 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.523622    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" containerName="revoke-keys"
Oct 30 00:57:52.523720 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.523711    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" containerName="revoke-keys"
Oct 30 00:57:52.607720 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.607677    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.616496 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.608108    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.616834 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.616801    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8vv\" (UniqueName: \"kubernetes.io/projected/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-kube-api-access-fl8vv\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.706099 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.664596    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.723720 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.723596    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:57:52.726919 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.726892    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:52.727196 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.727176    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:57:52.728043 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.727960    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:57:52.794893 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.794328    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.794893 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.794385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8vv\" (UniqueName: \"kubernetes.io/projected/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-kube-api-access-fl8vv\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.794893 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.794489    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.795609 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.795578    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.863903 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.862691    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:52.865442 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.865407    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"]
Oct 30 00:57:52.867076 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:52.866999    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8vv\" (UniqueName: \"kubernetes.io/projected/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-kube-api-access-fl8vv\") pod \"maas-api-revoke-sub-a18f3a6a-dv2pj\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:53.097902 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:53.097801    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:53.401115 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:53.401086    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"]
Oct 30 00:57:53.404198 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:57:53.404152    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce60ea83_54d1_4103_9d18_e3ed4b201ec7.slice/crio-c2c492e60f07228831c71d46c701ba7eb1262584f98601d29937d33e360ccb77 WatchSource:0}: Error finding container c2c492e60f07228831c71d46c701ba7eb1262584f98601d29937d33e360ccb77: Status 404 returned error can't find the container with id c2c492e60f07228831c71d46c701ba7eb1262584f98601d29937d33e360ccb77
Oct 30 00:57:53.882149 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:53.882100    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj" event={"ID":"ce60ea83-54d1-4103-9d18-e3ed4b201ec7","Type":"ContainerStarted","Data":"b672a8ef2347ca429015a5e70c5d23b71f432b02e41ed183186923d62fa73cf5"}
Oct 30 00:57:53.882674 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:53.882177    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj" event={"ID":"ce60ea83-54d1-4103-9d18-e3ed4b201ec7","Type":"ContainerStarted","Data":"c2c492e60f07228831c71d46c701ba7eb1262584f98601d29937d33e360ccb77"}
Oct 30 00:57:53.921608 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:53.921552    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj" podStartSLOduration=1.921531356 podStartE2EDuration="1.921531356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:53.917644456 +0000 UTC m=+915.465334952" watchObservedRunningTime="2026-10-30 00:57:53.921531356 +0000 UTC m=+915.469221851"
Oct 30 00:57:54.366858 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.366826    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"]
Oct 30 00:57:54.370877 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.370860    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.389908 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.389875    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"]
Oct 30 00:57:54.514233 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.514188    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wbp\" (UniqueName: \"kubernetes.io/projected/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-kube-api-access-99wbp\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.514411 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.514358    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.514551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.514520    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.615614 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.615523    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wbp\" (UniqueName: \"kubernetes.io/projected/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-kube-api-access-99wbp\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.615759 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.615659    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.615759 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.615751    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.616350 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.616322    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.618352 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.618319    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.634564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.634534    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wbp\" (UniqueName: \"kubernetes.io/projected/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-kube-api-access-99wbp\") pod \"maas-api-revoke-sub-6dbc9862-jmw4n\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.686633 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.686594    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:54.891583 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.891195    5104 generic.go:356] "Generic (PLEG): container finished" podID="ce60ea83-54d1-4103-9d18-e3ed4b201ec7" containerID="b672a8ef2347ca429015a5e70c5d23b71f432b02e41ed183186923d62fa73cf5" exitCode=0
Oct 30 00:57:54.891583 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.891293    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj" event={"ID":"ce60ea83-54d1-4103-9d18-e3ed4b201ec7","Type":"ContainerDied","Data":"b672a8ef2347ca429015a5e70c5d23b71f432b02e41ed183186923d62fa73cf5"}
Oct 30 00:57:54.992358 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:54.992315    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"]
Oct 30 00:57:54.995892 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:57:54.995846    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9f0a2d_d142_4da1_83b4_e4571c3c3755.slice/crio-eda77d3b87442b232658ae778b3015c5c41ccaf74cfaaf21d65be68e65031a87 WatchSource:0}: Error finding container eda77d3b87442b232658ae778b3015c5c41ccaf74cfaaf21d65be68e65031a87: Status 404 returned error can't find the container with id eda77d3b87442b232658ae778b3015c5c41ccaf74cfaaf21d65be68e65031a87
Oct 30 00:57:55.901163 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:55.901118    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n" event={"ID":"4a9f0a2d-d142-4da1-83b4-e4571c3c3755","Type":"ContainerStarted","Data":"ae7227ac9be50069428a4a593ca626d2d2de563955fa0cf238cbadcf1c13d851"}
Oct 30 00:57:55.901650 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:55.901187    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n" event={"ID":"4a9f0a2d-d142-4da1-83b4-e4571c3c3755","Type":"ContainerStarted","Data":"eda77d3b87442b232658ae778b3015c5c41ccaf74cfaaf21d65be68e65031a87"}
Oct 30 00:57:55.938726 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:55.938648    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n" podStartSLOduration=1.938627694 podStartE2EDuration="1.938627694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:55.938010327 +0000 UTC m=+917.485700824" watchObservedRunningTime="2026-10-30 00:57:55.938627694 +0000 UTC m=+917.486318191"
Oct 30 00:57:56.075593 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.075553    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:56.136174 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.136140    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-xks-ca\") pod \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") "
Oct 30 00:57:56.136599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.136191    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-kube-api-access-fl8vv\" (UniqueName: \"kubernetes.io/projected/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-kube-api-access-fl8vv\") pod \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") "
Oct 30 00:57:56.136599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.136232    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-openshift-ca\") pod \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\" (UID: \"ce60ea83-54d1-4103-9d18-e3ed4b201ec7\") "
Oct 30 00:57:56.137269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.137242    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-openshift-ca" pod "ce60ea83-54d1-4103-9d18-e3ed4b201ec7" (UID: "ce60ea83-54d1-4103-9d18-e3ed4b201ec7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:56.138723 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.138698    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-xks-ca" pod "ce60ea83-54d1-4103-9d18-e3ed4b201ec7" (UID: "ce60ea83-54d1-4103-9d18-e3ed4b201ec7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:56.139225 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.139197    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-kube-api-access-fl8vv" pod "ce60ea83-54d1-4103-9d18-e3ed4b201ec7" (UID: "ce60ea83-54d1-4103-9d18-e3ed4b201ec7"). InnerVolumeSpecName "kube-api-access-fl8vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:56.238074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.238030    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:56.238074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.238071    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl8vv\" (UniqueName: \"kubernetes.io/projected/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-kube-api-access-fl8vv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:56.238280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.238086    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce60ea83-54d1-4103-9d18-e3ed4b201ec7-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:56.910004 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.909914    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"
Oct 30 00:57:56.910004 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.909914    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj" event={"ID":"ce60ea83-54d1-4103-9d18-e3ed4b201ec7","Type":"ContainerDied","Data":"c2c492e60f07228831c71d46c701ba7eb1262584f98601d29937d33e360ccb77"}
Oct 30 00:57:56.910539 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.910058    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c492e60f07228831c71d46c701ba7eb1262584f98601d29937d33e360ccb77"
Oct 30 00:57:56.912089 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.912055    5104 generic.go:356] "Generic (PLEG): container finished" podID="4a9f0a2d-d142-4da1-83b4-e4571c3c3755" containerID="ae7227ac9be50069428a4a593ca626d2d2de563955fa0cf238cbadcf1c13d851" exitCode=0
Oct 30 00:57:56.912224 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:56.912202    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n" event={"ID":"4a9f0a2d-d142-4da1-83b4-e4571c3c3755","Type":"ContainerDied","Data":"ae7227ac9be50069428a4a593ca626d2d2de563955fa0cf238cbadcf1c13d851"}
Oct 30 00:57:58.082862 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.082829    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:58.170213 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.170103    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-openshift-ca\") pod \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") "
Oct 30 00:57:58.170213 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.170196    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-xks-ca\") pod \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") "
Oct 30 00:57:58.170498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.170245    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-kube-api-access-99wbp\" (UniqueName: \"kubernetes.io/projected/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-kube-api-access-99wbp\") pod \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\" (UID: \"4a9f0a2d-d142-4da1-83b4-e4571c3c3755\") "
Oct 30 00:57:58.170639 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.170610    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-openshift-ca" pod "4a9f0a2d-d142-4da1-83b4-e4571c3c3755" (UID: "4a9f0a2d-d142-4da1-83b4-e4571c3c3755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:58.170883 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.170864    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:58.172594 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.172566    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-kube-api-access-99wbp" pod "4a9f0a2d-d142-4da1-83b4-e4571c3c3755" (UID: "4a9f0a2d-d142-4da1-83b4-e4571c3c3755"). InnerVolumeSpecName "kube-api-access-99wbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:58.173121 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.173095    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-xks-ca" pod "4a9f0a2d-d142-4da1-83b4-e4571c3c3755" (UID: "4a9f0a2d-d142-4da1-83b4-e4571c3c3755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:58.271721 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.271682    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:58.271721 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.271716    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wbp\" (UniqueName: \"kubernetes.io/projected/4a9f0a2d-d142-4da1-83b4-e4571c3c3755-kube-api-access-99wbp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:57:58.926388 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.926353    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n" event={"ID":"4a9f0a2d-d142-4da1-83b4-e4571c3c3755","Type":"ContainerDied","Data":"eda77d3b87442b232658ae778b3015c5c41ccaf74cfaaf21d65be68e65031a87"}
Oct 30 00:57:58.926388 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.926385    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"
Oct 30 00:57:58.926619 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:57:58.926427    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda77d3b87442b232658ae778b3015c5c41ccaf74cfaaf21d65be68e65031a87"
Oct 30 00:58:00.315487 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.315439    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"]
Oct 30 00:58:00.316189 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.316169    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4a9f0a2d-d142-4da1-83b4-e4571c3c3755" containerName="revoke-keys"
Oct 30 00:58:00.316280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.316191    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9f0a2d-d142-4da1-83b4-e4571c3c3755" containerName="revoke-keys"
Oct 30 00:58:00.316280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.316225    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ce60ea83-54d1-4103-9d18-e3ed4b201ec7" containerName="revoke-keys"
Oct 30 00:58:00.316280 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.316231    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce60ea83-54d1-4103-9d18-e3ed4b201ec7" containerName="revoke-keys"
Oct 30 00:58:00.316376 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.316326    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9f0a2d-d142-4da1-83b4-e4571c3c3755" containerName="revoke-keys"
Oct 30 00:58:00.316376 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.316340    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce60ea83-54d1-4103-9d18-e3ed4b201ec7" containerName="revoke-keys"
Oct 30 00:58:00.320138 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.320120    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.345804 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.345775    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:58:00.346087 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.346004    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:00.346087 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.346050    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:58:00.346087 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.346074    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:58:00.349690 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.349663    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"]
Oct 30 00:58:00.405350 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.405316    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hxq\" (UniqueName: \"kubernetes.io/projected/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-kube-api-access-q7hxq\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.405560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.405473    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.405560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.405511    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.506722 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.506687    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.506903 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.506735    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.506903 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.506830    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hxq\" (UniqueName: \"kubernetes.io/projected/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-kube-api-access-q7hxq\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.507482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.507422    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.509129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.509102    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.525388 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.525360    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hxq\" (UniqueName: \"kubernetes.io/projected/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-kube-api-access-q7hxq\") pod \"maas-api-revoke-sub-c3ab9331-qrwmg\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.636421 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.636340    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:00.935624 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:00.935588    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"]
Oct 30 00:58:00.939998 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:00.939968    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a40d1e5_98a5_47a8_b1aa_5d5539709b34.slice/crio-4037ed6a6956af6d4a7bc472b3d9de6a567f287a4ae6f3e5a574f834fb1aed59 WatchSource:0}: Error finding container 4037ed6a6956af6d4a7bc472b3d9de6a567f287a4ae6f3e5a574f834fb1aed59: Status 404 returned error can't find the container with id 4037ed6a6956af6d4a7bc472b3d9de6a567f287a4ae6f3e5a574f834fb1aed59
Oct 30 00:58:01.054380 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.054342    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"]
Oct 30 00:58:01.060158 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.060136    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.074762 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.074737    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"]
Oct 30 00:58:01.115825 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.115789    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.116014 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.115852    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.116014 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.115968    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxsp\" (UniqueName: \"kubernetes.io/projected/62cb1040-f3ab-4784-b08d-e327832ef901-kube-api-access-wxxsp\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.217604 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.217504    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxsp\" (UniqueName: \"kubernetes.io/projected/62cb1040-f3ab-4784-b08d-e327832ef901-kube-api-access-wxxsp\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.217787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.217727    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.217865 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.217797    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.218403 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.218379    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.220186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.220162    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.235855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.235823    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxsp\" (UniqueName: \"kubernetes.io/projected/62cb1040-f3ab-4784-b08d-e327832ef901-kube-api-access-wxxsp\") pod \"maas-api-revoke-sub-4eb340ac-nvrhf\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.375499 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.375466    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:01.685658 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.685617    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"]
Oct 30 00:58:01.705816 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:01.705681    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cb1040_f3ab_4784_b08d_e327832ef901.slice/crio-b5195287a9e736106dadd6e363f87d160346dd36d8c3d8346b3ec04806759320 WatchSource:0}: Error finding container b5195287a9e736106dadd6e363f87d160346dd36d8c3d8346b3ec04806759320: Status 404 returned error can't find the container with id b5195287a9e736106dadd6e363f87d160346dd36d8c3d8346b3ec04806759320
Oct 30 00:58:01.951605 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.951495    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf" event={"ID":"62cb1040-f3ab-4784-b08d-e327832ef901","Type":"ContainerStarted","Data":"b5195287a9e736106dadd6e363f87d160346dd36d8c3d8346b3ec04806759320"}
Oct 30 00:58:01.953537 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.953505    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg" event={"ID":"9a40d1e5-98a5-47a8-b1aa-5d5539709b34","Type":"ContainerStarted","Data":"877ed95cdc0ab85ae2a9119736af0e696595d478f1006b19ab16ef8c31046b58"}
Oct 30 00:58:01.953671 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.953557    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg" event={"ID":"9a40d1e5-98a5-47a8-b1aa-5d5539709b34","Type":"ContainerStarted","Data":"4037ed6a6956af6d4a7bc472b3d9de6a567f287a4ae6f3e5a574f834fb1aed59"}
Oct 30 00:58:01.992883 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:01.992832    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg" podStartSLOduration=1.9928168149999999 podStartE2EDuration="1.992816815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:01.990687175 +0000 UTC m=+923.538377678" watchObservedRunningTime="2026-10-30 00:58:01.992816815 +0000 UTC m=+923.540507352"
Oct 30 00:58:02.152250 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.152207    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"]
Oct 30 00:58:02.156401 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.156376    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.173635 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.173603    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"]
Oct 30 00:58:02.230400 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.230316    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.230580 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.230403    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.230625 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.230572    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pmwq\" (UniqueName: \"kubernetes.io/projected/ff1548dd-6b73-479c-9060-ee13609ade59-kube-api-access-4pmwq\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.331721 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.331677    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pmwq\" (UniqueName: \"kubernetes.io/projected/ff1548dd-6b73-479c-9060-ee13609ade59-kube-api-access-4pmwq\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.331876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.331770    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.331876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.331833    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.332639 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.332610    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.334297 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.334263    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.349809 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.349780    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pmwq\" (UniqueName: \"kubernetes.io/projected/ff1548dd-6b73-479c-9060-ee13609ade59-kube-api-access-4pmwq\") pod \"maas-api-revoke-sub-a25de51f-742cq\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.470072 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.470039    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:02.777992 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.777956    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"]
Oct 30 00:58:02.780049 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:02.780018    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1548dd_6b73_479c_9060_ee13609ade59.slice/crio-fda3b717b362e9ffca47200946ff7bbe4d19f6f84ae4c93c63a2f4dda1a6dbbe WatchSource:0}: Error finding container fda3b717b362e9ffca47200946ff7bbe4d19f6f84ae4c93c63a2f4dda1a6dbbe: Status 404 returned error can't find the container with id fda3b717b362e9ffca47200946ff7bbe4d19f6f84ae4c93c63a2f4dda1a6dbbe
Oct 30 00:58:02.963753 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.963714    5104 generic.go:356] "Generic (PLEG): container finished" podID="62cb1040-f3ab-4784-b08d-e327832ef901" containerID="7c6e33216ccfa3b4af66f0baccdb2f9a26fcf1fe3d650562a9d65504bb0594f7" exitCode=0
Oct 30 00:58:02.963925 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.963799    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf" event={"ID":"62cb1040-f3ab-4784-b08d-e327832ef901","Type":"ContainerDied","Data":"7c6e33216ccfa3b4af66f0baccdb2f9a26fcf1fe3d650562a9d65504bb0594f7"}
Oct 30 00:58:02.965229 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.965195    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq" event={"ID":"ff1548dd-6b73-479c-9060-ee13609ade59","Type":"ContainerStarted","Data":"fda3b717b362e9ffca47200946ff7bbe4d19f6f84ae4c93c63a2f4dda1a6dbbe"}
Oct 30 00:58:02.966786 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.966759    5104 generic.go:356] "Generic (PLEG): container finished" podID="9a40d1e5-98a5-47a8-b1aa-5d5539709b34" containerID="877ed95cdc0ab85ae2a9119736af0e696595d478f1006b19ab16ef8c31046b58" exitCode=0
Oct 30 00:58:02.966909 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:02.966825    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg" event={"ID":"9a40d1e5-98a5-47a8-b1aa-5d5539709b34","Type":"ContainerDied","Data":"877ed95cdc0ab85ae2a9119736af0e696595d478f1006b19ab16ef8c31046b58"}
Oct 30 00:58:03.976930 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:03.976827    5104 generic.go:356] "Generic (PLEG): container finished" podID="ff1548dd-6b73-479c-9060-ee13609ade59" containerID="0d645330c2c281d30a23e3b6584321e663b19e63b353770b257c43719d45ad67" exitCode=0
Oct 30 00:58:03.976930 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:03.976907    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq" event={"ID":"ff1548dd-6b73-479c-9060-ee13609ade59","Type":"ContainerDied","Data":"0d645330c2c281d30a23e3b6584321e663b19e63b353770b257c43719d45ad67"}
Oct 30 00:58:04.178851 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.178824    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:04.212703 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.212679    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:04.260159 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.260067    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-openshift-ca\") pod \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") "
Oct 30 00:58:04.260159 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.260126    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-kube-api-access-q7hxq\" (UniqueName: \"kubernetes.io/projected/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-kube-api-access-q7hxq\") pod \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") "
Oct 30 00:58:04.260371 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.260226    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-openshift-ca\") pod \"62cb1040-f3ab-4784-b08d-e327832ef901\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") "
Oct 30 00:58:04.260371 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.260253    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cb1040-f3ab-4784-b08d-e327832ef901-kube-api-access-wxxsp\" (UniqueName: \"kubernetes.io/projected/62cb1040-f3ab-4784-b08d-e327832ef901-kube-api-access-wxxsp\") pod \"62cb1040-f3ab-4784-b08d-e327832ef901\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") "
Oct 30 00:58:04.260371 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.260294    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-xks-ca\") pod \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\" (UID: \"9a40d1e5-98a5-47a8-b1aa-5d5539709b34\") "
Oct 30 00:58:04.260603 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.260383    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-xks-ca\") pod \"62cb1040-f3ab-4784-b08d-e327832ef901\" (UID: \"62cb1040-f3ab-4784-b08d-e327832ef901\") "
Oct 30 00:58:04.261709 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.261676    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-openshift-ca" pod "62cb1040-f3ab-4784-b08d-e327832ef901" (UID: "62cb1040-f3ab-4784-b08d-e327832ef901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:04.261830 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.261681    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-openshift-ca" pod "9a40d1e5-98a5-47a8-b1aa-5d5539709b34" (UID: "9a40d1e5-98a5-47a8-b1aa-5d5539709b34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:04.263898 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.263861    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-xks-ca" pod "62cb1040-f3ab-4784-b08d-e327832ef901" (UID: "62cb1040-f3ab-4784-b08d-e327832ef901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:04.264361 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.264333    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cb1040-f3ab-4784-b08d-e327832ef901-kube-api-access-wxxsp" pod "62cb1040-f3ab-4784-b08d-e327832ef901" (UID: "62cb1040-f3ab-4784-b08d-e327832ef901"). InnerVolumeSpecName "kube-api-access-wxxsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:04.264979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.264956    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-xks-ca" pod "9a40d1e5-98a5-47a8-b1aa-5d5539709b34" (UID: "9a40d1e5-98a5-47a8-b1aa-5d5539709b34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:04.265068 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.265001    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-kube-api-access-q7hxq" pod "9a40d1e5-98a5-47a8-b1aa-5d5539709b34" (UID: "9a40d1e5-98a5-47a8-b1aa-5d5539709b34"). InnerVolumeSpecName "kube-api-access-q7hxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:04.362553 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.362518    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:04.362553 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.362549    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:04.362553 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.362561    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hxq\" (UniqueName: \"kubernetes.io/projected/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-kube-api-access-q7hxq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:04.362784 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.362570    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cb1040-f3ab-4784-b08d-e327832ef901-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:04.362784 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.362579    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxxsp\" (UniqueName: \"kubernetes.io/projected/62cb1040-f3ab-4784-b08d-e327832ef901-kube-api-access-wxxsp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:04.362784 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.362589    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a40d1e5-98a5-47a8-b1aa-5d5539709b34-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:04.985395 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.985357    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"
Oct 30 00:58:04.985395 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.985370    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf" event={"ID":"62cb1040-f3ab-4784-b08d-e327832ef901","Type":"ContainerDied","Data":"b5195287a9e736106dadd6e363f87d160346dd36d8c3d8346b3ec04806759320"}
Oct 30 00:58:04.985968 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.985432    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5195287a9e736106dadd6e363f87d160346dd36d8c3d8346b3ec04806759320"
Oct 30 00:58:04.987093 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.987064    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"
Oct 30 00:58:04.987239 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.987107    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg" event={"ID":"9a40d1e5-98a5-47a8-b1aa-5d5539709b34","Type":"ContainerDied","Data":"4037ed6a6956af6d4a7bc472b3d9de6a567f287a4ae6f3e5a574f834fb1aed59"}
Oct 30 00:58:04.987239 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:04.987163    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4037ed6a6956af6d4a7bc472b3d9de6a567f287a4ae6f3e5a574f834fb1aed59"
Oct 30 00:58:05.197265 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.197237    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:05.275787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.275295    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-xks-ca\") pod \"ff1548dd-6b73-479c-9060-ee13609ade59\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") "
Oct 30 00:58:05.275787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.275390    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1548dd-6b73-479c-9060-ee13609ade59-kube-api-access-4pmwq\" (UniqueName: \"kubernetes.io/projected/ff1548dd-6b73-479c-9060-ee13609ade59-kube-api-access-4pmwq\") pod \"ff1548dd-6b73-479c-9060-ee13609ade59\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") "
Oct 30 00:58:05.275787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.275495    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-openshift-ca\") pod \"ff1548dd-6b73-479c-9060-ee13609ade59\" (UID: \"ff1548dd-6b73-479c-9060-ee13609ade59\") "
Oct 30 00:58:05.276389 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.276355    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-openshift-ca" pod "ff1548dd-6b73-479c-9060-ee13609ade59" (UID: "ff1548dd-6b73-479c-9060-ee13609ade59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:05.278348 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.278314    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-xks-ca" pod "ff1548dd-6b73-479c-9060-ee13609ade59" (UID: "ff1548dd-6b73-479c-9060-ee13609ade59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:05.278764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.278738    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1548dd-6b73-479c-9060-ee13609ade59-kube-api-access-4pmwq" pod "ff1548dd-6b73-479c-9060-ee13609ade59" (UID: "ff1548dd-6b73-479c-9060-ee13609ade59"). InnerVolumeSpecName "kube-api-access-4pmwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:05.377554 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.377520    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:05.377554 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.377556    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pmwq\" (UniqueName: \"kubernetes.io/projected/ff1548dd-6b73-479c-9060-ee13609ade59-kube-api-access-4pmwq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:05.377759 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:05.377568    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1548dd-6b73-479c-9060-ee13609ade59-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:06.011962 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:06.011922    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"
Oct 30 00:58:06.011962 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:06.011945    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq" event={"ID":"ff1548dd-6b73-479c-9060-ee13609ade59","Type":"ContainerDied","Data":"fda3b717b362e9ffca47200946ff7bbe4d19f6f84ae4c93c63a2f4dda1a6dbbe"}
Oct 30 00:58:06.012381 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:06.011992    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fda3b717b362e9ffca47200946ff7bbe4d19f6f84ae4c93c63a2f4dda1a6dbbe"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.906766    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"]
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907688    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9a40d1e5-98a5-47a8-b1aa-5d5539709b34" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907706    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a40d1e5-98a5-47a8-b1aa-5d5539709b34" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907749    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ff1548dd-6b73-479c-9060-ee13609ade59" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907758    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1548dd-6b73-479c-9060-ee13609ade59" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907772    5104 cpu_manager.go:414] "containerMap: removing container" podUID="62cb1040-f3ab-4784-b08d-e327832ef901" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907780    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cb1040-f3ab-4784-b08d-e327832ef901" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907904    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cb1040-f3ab-4784-b08d-e327832ef901" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907922    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a40d1e5-98a5-47a8-b1aa-5d5539709b34" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.907936    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1548dd-6b73-479c-9060-ee13609ade59" containerName="revoke-keys"
Oct 30 00:58:07.913522 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.912390    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:07.933148 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.933102    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"]
Oct 30 00:58:07.942674 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.941000    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:58:07.942674 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.941210    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:58:07.942674 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.941321    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:07.942674 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.941423    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:58:07.956269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.956219    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:07.956477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.956392    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:07.956570 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:07.956484    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q24zs\" (UniqueName: \"kubernetes.io/projected/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-kube-api-access-q24zs\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.058482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.057649    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.058482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.057816    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.058482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.057890    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24zs\" (UniqueName: \"kubernetes.io/projected/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-kube-api-access-q24zs\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.062480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.059145    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.069513 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.064865    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.081060 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.081026    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24zs\" (UniqueName: \"kubernetes.io/projected/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-kube-api-access-q24zs\") pod \"maas-api-revoke-sub-8ec74fc3-4bkfb\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.260256 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.260223    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:08.867532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:08.867022    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"]
Oct 30 00:58:09.133540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:09.133066    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb" event={"ID":"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b","Type":"ContainerStarted","Data":"95ab2fd9b31be8938b7be846228dc9faccbae7b8d431caa450b0c1a73afc20ef"}
Oct 30 00:58:09.303484 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:58:09.298681    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:58:10.328518 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:10.321343    5104 generic.go:356] "Generic (PLEG): container finished" podID="07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" containerID="f5328cb72ad56eef6b4b69968ed88ddfea17130c8ec96cf8bd94f9a2948769fd" exitCode=0
Oct 30 00:58:10.330792 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:10.329273    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb" event={"ID":"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b","Type":"ContainerDied","Data":"f5328cb72ad56eef6b4b69968ed88ddfea17130c8ec96cf8bd94f9a2948769fd"}
Oct 30 00:58:11.509232 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.509195    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"]
Oct 30 00:58:11.513961 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.513931    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:11.514840 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.514821    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.542705 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.542679    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"]
Oct 30 00:58:11.553812 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.553787    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-openshift-ca\") pod \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") "
Oct 30 00:58:11.553963 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.553850    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-xks-ca\") pod \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") "
Oct 30 00:58:11.553963 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.553950    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-kube-api-access-q24zs\" (UniqueName: \"kubernetes.io/projected/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-kube-api-access-q24zs\") pod \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\" (UID: \"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b\") "
Oct 30 00:58:11.554257 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.554214    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-openshift-ca" pod "07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" (UID: "07bdc66c-fdaf-4e19-be14-58b9cdacbc5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:11.554257 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.554225    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.554558 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.554530    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.554655 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.554600    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2j5\" (UniqueName: \"kubernetes.io/projected/5bfc690e-c9f9-4c5a-9518-6668feda534c-kube-api-access-jd2j5\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.554804 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.554779    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:11.556252 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.556225    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-kube-api-access-q24zs" pod "07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" (UID: "07bdc66c-fdaf-4e19-be14-58b9cdacbc5b"). InnerVolumeSpecName "kube-api-access-q24zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:11.556331 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.556296    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-xks-ca" pod "07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" (UID: "07bdc66c-fdaf-4e19-be14-58b9cdacbc5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:11.655628 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.655542    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.655628 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.655605    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2j5\" (UniqueName: \"kubernetes.io/projected/5bfc690e-c9f9-4c5a-9518-6668feda534c-kube-api-access-jd2j5\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.655843 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.655682    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.655843 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.655819    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:11.655843 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.655831    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q24zs\" (UniqueName: \"kubernetes.io/projected/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b-kube-api-access-q24zs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:11.656398 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.656377    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.658099 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.658076    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.673520 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.673493    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2j5\" (UniqueName: \"kubernetes.io/projected/5bfc690e-c9f9-4c5a-9518-6668feda534c-kube-api-access-jd2j5\") pod \"maas-api-revoke-sub-858cc330-gxdct\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:11.831583 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:11.831544    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:12.134682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:12.134652    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"]
Oct 30 00:58:12.136485 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:12.136439    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfc690e_c9f9_4c5a_9518_6668feda534c.slice/crio-ffa45fbdb6ef002929753966f8bb10cafe25a3e62ca485dcf72267cbe9cee9fb WatchSource:0}: Error finding container ffa45fbdb6ef002929753966f8bb10cafe25a3e62ca485dcf72267cbe9cee9fb: Status 404 returned error can't find the container with id ffa45fbdb6ef002929753966f8bb10cafe25a3e62ca485dcf72267cbe9cee9fb
Oct 30 00:58:12.344288 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:12.344248    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct" event={"ID":"5bfc690e-c9f9-4c5a-9518-6668feda534c","Type":"ContainerStarted","Data":"ffa45fbdb6ef002929753966f8bb10cafe25a3e62ca485dcf72267cbe9cee9fb"}
Oct 30 00:58:12.345820 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:12.345791    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"
Oct 30 00:58:12.345820 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:12.345810    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb" event={"ID":"07bdc66c-fdaf-4e19-be14-58b9cdacbc5b","Type":"ContainerDied","Data":"95ab2fd9b31be8938b7be846228dc9faccbae7b8d431caa450b0c1a73afc20ef"}
Oct 30 00:58:12.345990 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:12.345859    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95ab2fd9b31be8938b7be846228dc9faccbae7b8d431caa450b0c1a73afc20ef"
Oct 30 00:58:13.277051 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.276964    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"]
Oct 30 00:58:13.277589 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.277571    5104 cpu_manager.go:414] "containerMap: removing container" podUID="07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" containerName="revoke-keys"
Oct 30 00:58:13.277589 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.277589    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" containerName="revoke-keys"
Oct 30 00:58:13.277718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.277693    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" containerName="revoke-keys"
Oct 30 00:58:13.283388 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.283347    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.298836 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.298764    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"]
Oct 30 00:58:13.353792 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.353758    5104 generic.go:356] "Generic (PLEG): container finished" podID="5bfc690e-c9f9-4c5a-9518-6668feda534c" containerID="7151ec6167b5a0b56cd11b3d1fc9bc6bbb2fc422e5673ef3f467a7a499157c0c" exitCode=0
Oct 30 00:58:13.353948 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.353797    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct" event={"ID":"5bfc690e-c9f9-4c5a-9518-6668feda534c","Type":"ContainerDied","Data":"7151ec6167b5a0b56cd11b3d1fc9bc6bbb2fc422e5673ef3f467a7a499157c0c"}
Oct 30 00:58:13.376032 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.375997    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4h28\" (UniqueName: \"kubernetes.io/projected/8fdf9d33-647e-460b-9186-5ea2c337078a-kube-api-access-l4h28\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.376234 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.376216    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.376315 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.376301    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.477638 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.477594    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4h28\" (UniqueName: \"kubernetes.io/projected/8fdf9d33-647e-460b-9186-5ea2c337078a-kube-api-access-l4h28\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.477852 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.477743    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.477852 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.477785    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.478439 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.478413    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.480204 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.480181    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.496442 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.496408    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4h28\" (UniqueName: \"kubernetes.io/projected/8fdf9d33-647e-460b-9186-5ea2c337078a-kube-api-access-l4h28\") pod \"maas-api-revoke-sub-d97829e2-x5r6h\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.599913 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.599832    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:13.907103 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:13.907076    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"]
Oct 30 00:58:13.909258 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:13.909226    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdf9d33_647e_460b_9186_5ea2c337078a.slice/crio-fea181dd2e7a2222118c89a9c6a86111bd485fa64bcecf5460500f84fa5660db WatchSource:0}: Error finding container fea181dd2e7a2222118c89a9c6a86111bd485fa64bcecf5460500f84fa5660db: Status 404 returned error can't find the container with id fea181dd2e7a2222118c89a9c6a86111bd485fa64bcecf5460500f84fa5660db
Oct 30 00:58:14.221283 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.221243    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"]
Oct 30 00:58:14.229129 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.229098    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.252054 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.252017    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"distinct-w4-d827b7-kserve-self-signed-certs\""
Oct 30 00:58:14.253305 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.253270    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"]
Oct 30 00:58:14.290564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.290517    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-tmp-dir\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.290999 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.290663    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-dshm\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.290999 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.290699    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-model-cache\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.290999 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.290787    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrz2n\" (UniqueName: \"kubernetes.io/projected/2a302533-defe-4800-ae63-74070ee988a5-kube-api-access-wrz2n\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.290999 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.290815    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a302533-defe-4800-ae63-74070ee988a5-tls-certs\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.290999 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.290891    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-home\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.363833 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.363795    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h" event={"ID":"8fdf9d33-647e-460b-9186-5ea2c337078a","Type":"ContainerStarted","Data":"53f350f0d3c39bcd7b7256d3d00e6755dd692f258534e69b51b858c4c593e98d"}
Oct 30 00:58:14.364041 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.363865    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h" event={"ID":"8fdf9d33-647e-460b-9186-5ea2c337078a","Type":"ContainerStarted","Data":"fea181dd2e7a2222118c89a9c6a86111bd485fa64bcecf5460500f84fa5660db"}
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.392214    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrz2n\" (UniqueName: \"kubernetes.io/projected/2a302533-defe-4800-ae63-74070ee988a5-kube-api-access-wrz2n\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.392262    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a302533-defe-4800-ae63-74070ee988a5-tls-certs\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.392398    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-home\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.392561    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-tmp-dir\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.392649    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-dshm\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.392673    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-model-cache\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.393133    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-model-cache\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.394489    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-home\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.395261    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-tmp-dir\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.398485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.398207    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-dshm\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.401065 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.401037    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a302533-defe-4800-ae63-74070ee988a5-tls-certs\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.409356 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.409287    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h" podStartSLOduration=1.409274135 podStartE2EDuration="1.409274135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:14.406149933 +0000 UTC m=+935.953840430" watchObservedRunningTime="2026-10-30 00:58:14.409274135 +0000 UTC m=+935.956964631"
Oct 30 00:58:14.412802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.412127    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrz2n\" (UniqueName: \"kubernetes.io/projected/2a302533-defe-4800-ae63-74070ee988a5-kube-api-access-wrz2n\") pod \"distinct-w4-d827b7-kserve-564ccc6cb9-48kzv\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.574526 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.572959    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:14.641602 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.641567    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:14.715428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.712891    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-xks-ca\") pod \"5bfc690e-c9f9-4c5a-9518-6668feda534c\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") "
Oct 30 00:58:14.715428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.713396    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-openshift-ca\") pod \"5bfc690e-c9f9-4c5a-9518-6668feda534c\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") "
Oct 30 00:58:14.715428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.713789    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-openshift-ca" pod "5bfc690e-c9f9-4c5a-9518-6668feda534c" (UID: "5bfc690e-c9f9-4c5a-9518-6668feda534c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:14.715428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.713915    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfc690e-c9f9-4c5a-9518-6668feda534c-kube-api-access-jd2j5\" (UniqueName: \"kubernetes.io/projected/5bfc690e-c9f9-4c5a-9518-6668feda534c-kube-api-access-jd2j5\") pod \"5bfc690e-c9f9-4c5a-9518-6668feda534c\" (UID: \"5bfc690e-c9f9-4c5a-9518-6668feda534c\") "
Oct 30 00:58:14.715428 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.715374    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:14.717749 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.717706    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfc690e-c9f9-4c5a-9518-6668feda534c-kube-api-access-jd2j5" pod "5bfc690e-c9f9-4c5a-9518-6668feda534c" (UID: "5bfc690e-c9f9-4c5a-9518-6668feda534c"). InnerVolumeSpecName "kube-api-access-jd2j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:14.720418 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.720380    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-xks-ca" pod "5bfc690e-c9f9-4c5a-9518-6668feda534c" (UID: "5bfc690e-c9f9-4c5a-9518-6668feda534c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:14.816402 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.816367    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfc690e-c9f9-4c5a-9518-6668feda534c-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:14.816402 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.816398    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2j5\" (UniqueName: \"kubernetes.io/projected/5bfc690e-c9f9-4c5a-9518-6668feda534c-kube-api-access-jd2j5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:14.901881 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:14.901854    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"]
Oct 30 00:58:14.922469 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:14.922431    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a302533_defe_4800_ae63_74070ee988a5.slice/crio-5e817f3d76ab8a92041fd1e8aae5663802cabcb61698a1a868a30420b0e07700 WatchSource:0}: Error finding container 5e817f3d76ab8a92041fd1e8aae5663802cabcb61698a1a868a30420b0e07700: Status 404 returned error can't find the container with id 5e817f3d76ab8a92041fd1e8aae5663802cabcb61698a1a868a30420b0e07700
Oct 30 00:58:15.372561 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.372521    5104 generic.go:356] "Generic (PLEG): container finished" podID="8fdf9d33-647e-460b-9186-5ea2c337078a" containerID="53f350f0d3c39bcd7b7256d3d00e6755dd692f258534e69b51b858c4c593e98d" exitCode=0
Oct 30 00:58:15.373046 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.372616    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h" event={"ID":"8fdf9d33-647e-460b-9186-5ea2c337078a","Type":"ContainerDied","Data":"53f350f0d3c39bcd7b7256d3d00e6755dd692f258534e69b51b858c4c593e98d"}
Oct 30 00:58:15.375637 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.375603    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" event={"ID":"2a302533-defe-4800-ae63-74070ee988a5","Type":"ContainerStarted","Data":"0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb"}
Oct 30 00:58:15.375817 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.375651    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" event={"ID":"2a302533-defe-4800-ae63-74070ee988a5","Type":"ContainerStarted","Data":"5e817f3d76ab8a92041fd1e8aae5663802cabcb61698a1a868a30420b0e07700"}
Oct 30 00:58:15.377300 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.377265    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct" event={"ID":"5bfc690e-c9f9-4c5a-9518-6668feda534c","Type":"ContainerDied","Data":"ffa45fbdb6ef002929753966f8bb10cafe25a3e62ca485dcf72267cbe9cee9fb"}
Oct 30 00:58:15.377300 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.377296    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"
Oct 30 00:58:15.377485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.377312    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa45fbdb6ef002929753966f8bb10cafe25a3e62ca485dcf72267cbe9cee9fb"
Oct 30 00:58:15.449362 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.449296    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podStartSLOduration=1.449275601 podStartE2EDuration="1.449275601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:15.444652611 +0000 UTC m=+936.992343149" watchObservedRunningTime="2026-10-30 00:58:15.449275601 +0000 UTC m=+936.996966097"
Oct 30 00:58:15.576479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:15.575565    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:16.548411 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.548386    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:16.637909 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.637808    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-openshift-ca\") pod \"8fdf9d33-647e-460b-9186-5ea2c337078a\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") "
Oct 30 00:58:16.638074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.637997    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdf9d33-647e-460b-9186-5ea2c337078a-kube-api-access-l4h28\" (UniqueName: \"kubernetes.io/projected/8fdf9d33-647e-460b-9186-5ea2c337078a-kube-api-access-l4h28\") pod \"8fdf9d33-647e-460b-9186-5ea2c337078a\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") "
Oct 30 00:58:16.638074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.638024    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-xks-ca\") pod \"8fdf9d33-647e-460b-9186-5ea2c337078a\" (UID: \"8fdf9d33-647e-460b-9186-5ea2c337078a\") "
Oct 30 00:58:16.638283 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.638256    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-openshift-ca" pod "8fdf9d33-647e-460b-9186-5ea2c337078a" (UID: "8fdf9d33-647e-460b-9186-5ea2c337078a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:16.638689 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.638658    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:16.640438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.640412    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdf9d33-647e-460b-9186-5ea2c337078a-kube-api-access-l4h28" pod "8fdf9d33-647e-460b-9186-5ea2c337078a" (UID: "8fdf9d33-647e-460b-9186-5ea2c337078a"). InnerVolumeSpecName "kube-api-access-l4h28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:16.640729 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.640698    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-xks-ca" pod "8fdf9d33-647e-460b-9186-5ea2c337078a" (UID: "8fdf9d33-647e-460b-9186-5ea2c337078a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:16.747714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.747671    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4h28\" (UniqueName: \"kubernetes.io/projected/8fdf9d33-647e-460b-9186-5ea2c337078a-kube-api-access-l4h28\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:16.747714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:16.747713    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdf9d33-647e-460b-9186-5ea2c337078a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:17.392117 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:17.392083    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"
Oct 30 00:58:17.392117 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:17.392105    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h" event={"ID":"8fdf9d33-647e-460b-9186-5ea2c337078a","Type":"ContainerDied","Data":"fea181dd2e7a2222118c89a9c6a86111bd485fa64bcecf5460500f84fa5660db"}
Oct 30 00:58:17.392395 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:17.392205    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fea181dd2e7a2222118c89a9c6a86111bd485fa64bcecf5460500f84fa5660db"
Oct 30 00:58:20.247504 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:20.247382    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:58:22.539310 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.539271    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"]
Oct 30 00:58:22.572489 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.547497    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5bfc690e-c9f9-4c5a-9518-6668feda534c" containerName="revoke-keys"
Oct 30 00:58:22.675565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.675517    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfc690e-c9f9-4c5a-9518-6668feda534c" containerName="revoke-keys"
Oct 30 00:58:22.675832 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.675812    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8fdf9d33-647e-460b-9186-5ea2c337078a" containerName="revoke-keys"
Oct 30 00:58:22.675933 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.675920    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdf9d33-647e-460b-9186-5ea2c337078a" containerName="revoke-keys"
Oct 30 00:58:22.697164 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.697127    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdf9d33-647e-460b-9186-5ea2c337078a" containerName="revoke-keys"
Oct 30 00:58:22.697707 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.697685    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfc690e-c9f9-4c5a-9518-6668feda534c" containerName="revoke-keys"
Oct 30 00:58:22.735482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.702540    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"]
Oct 30 00:58:22.740630 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.740595    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:22.759009 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.758980    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:22.771504 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.770500    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:58:22.771504 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.770581    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:58:22.771504 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.770657    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:58:22.904523 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.898329    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4lwb\" (UniqueName: \"kubernetes.io/projected/c2122562-9974-47ec-8143-039fdb5cb303-kube-api-access-p4lwb\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:22.904523 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.898404    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:22.928739 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:22.928628    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:23.073481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:23.065123    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:23.073481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:23.065324    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4lwb\" (UniqueName: \"kubernetes.io/projected/c2122562-9974-47ec-8143-039fdb5cb303-kube-api-access-p4lwb\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:23.073481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:23.065389    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:23.073481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:23.066292    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:23.085476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:23.081891    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:23.142536 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:23.141273    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4lwb\" (UniqueName: \"kubernetes.io/projected/c2122562-9974-47ec-8143-039fdb5cb303-kube-api-access-p4lwb\") pod \"maas-api-revoke-sub-aba9680f-qj6sh\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:23.213503 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:23.212672    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:24.369095 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:24.366371    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"]
Oct 30 00:58:24.399790 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:24.399753    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2122562_9974_47ec_8143_039fdb5cb303.slice/crio-bd0d7549dc9cc2ea81e951ddf7e53ffa30896c025d13cd679efa8782dcaf090f WatchSource:0}: Error finding container bd0d7549dc9cc2ea81e951ddf7e53ffa30896c025d13cd679efa8782dcaf090f: Status 404 returned error can't find the container with id bd0d7549dc9cc2ea81e951ddf7e53ffa30896c025d13cd679efa8782dcaf090f
Oct 30 00:58:24.633531 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:24.622673    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:24.957982 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:24.957946    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:24.980017 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:24.979974    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh" event={"ID":"c2122562-9974-47ec-8143-039fdb5cb303","Type":"ContainerStarted","Data":"bd0d7549dc9cc2ea81e951ddf7e53ffa30896c025d13cd679efa8782dcaf090f"}
Oct 30 00:58:25.587979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:25.587876    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:58:25.989887 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:25.989855    5104 generic.go:356] "Generic (PLEG): container finished" podID="c2122562-9974-47ec-8143-039fdb5cb303" containerID="6c3de3a419b765e4912586a43aef50f15629dfc8798c191a544e3ec0e0b2f7ae" exitCode=0
Oct 30 00:58:25.990062 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:25.989931    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh" event={"ID":"c2122562-9974-47ec-8143-039fdb5cb303","Type":"ContainerDied","Data":"6c3de3a419b765e4912586a43aef50f15629dfc8798c191a544e3ec0e0b2f7ae"}
Oct 30 00:58:26.526168 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.526128    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"]
Oct 30 00:58:26.530354 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.530336    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.547888 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.547860    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"]
Oct 30 00:58:26.654904 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.654866    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.654904 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.654911    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.655328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.654996    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5646\" (UniqueName: \"kubernetes.io/projected/ef1482d1-eb55-4458-89ca-b766d00d313a-kube-api-access-z5646\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.756021 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.755982    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.756200 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.756032    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.756200 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.756078    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5646\" (UniqueName: \"kubernetes.io/projected/ef1482d1-eb55-4458-89ca-b766d00d313a-kube-api-access-z5646\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.756788 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.756767    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.758321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.758297    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.774166 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.774137    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5646\" (UniqueName: \"kubernetes.io/projected/ef1482d1-eb55-4458-89ca-b766d00d313a-kube-api-access-z5646\") pod \"maas-api-revoke-sub-efbbaded-nrs4s\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:26.844219 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:26.844134    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:27.156217 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.156188    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"]
Oct 30 00:58:27.272194 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.272171    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:27.393605 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.393580    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2122562-9974-47ec-8143-039fdb5cb303-kube-api-access-p4lwb\" (UniqueName: \"kubernetes.io/projected/c2122562-9974-47ec-8143-039fdb5cb303-kube-api-access-p4lwb\") pod \"c2122562-9974-47ec-8143-039fdb5cb303\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") "
Oct 30 00:58:27.393742 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.393675    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-openshift-ca\") pod \"c2122562-9974-47ec-8143-039fdb5cb303\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") "
Oct 30 00:58:27.393742 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.393704    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-xks-ca\") pod \"c2122562-9974-47ec-8143-039fdb5cb303\" (UID: \"c2122562-9974-47ec-8143-039fdb5cb303\") "
Oct 30 00:58:27.394028 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.394000    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-openshift-ca" pod "c2122562-9974-47ec-8143-039fdb5cb303" (UID: "c2122562-9974-47ec-8143-039fdb5cb303"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:27.395814 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.395790    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2122562-9974-47ec-8143-039fdb5cb303-kube-api-access-p4lwb" pod "c2122562-9974-47ec-8143-039fdb5cb303" (UID: "c2122562-9974-47ec-8143-039fdb5cb303"). InnerVolumeSpecName "kube-api-access-p4lwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:27.395910 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.395869    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-xks-ca" pod "c2122562-9974-47ec-8143-039fdb5cb303" (UID: "c2122562-9974-47ec-8143-039fdb5cb303"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:27.494787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.494763    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4lwb\" (UniqueName: \"kubernetes.io/projected/c2122562-9974-47ec-8143-039fdb5cb303-kube-api-access-p4lwb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:27.494787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.494787    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:27.494943 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:27.494797    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2122562-9974-47ec-8143-039fdb5cb303-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:28.006589 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:28.006560    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"
Oct 30 00:58:28.007002 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:28.006593    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh" event={"ID":"c2122562-9974-47ec-8143-039fdb5cb303","Type":"ContainerDied","Data":"bd0d7549dc9cc2ea81e951ddf7e53ffa30896c025d13cd679efa8782dcaf090f"}
Oct 30 00:58:28.007002 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:28.006639    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd0d7549dc9cc2ea81e951ddf7e53ffa30896c025d13cd679efa8782dcaf090f"
Oct 30 00:58:28.008549 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:28.008524    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s" event={"ID":"ef1482d1-eb55-4458-89ca-b766d00d313a","Type":"ContainerStarted","Data":"08919208f41897f733481de3b620133c67a16796292288524135cb07b2855a5f"}
Oct 30 00:58:28.008683 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:28.008570    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s" event={"ID":"ef1482d1-eb55-4458-89ca-b766d00d313a","Type":"ContainerStarted","Data":"03cf21258a4c5ee75b33f74266f5d99a8a15133b3aaaf3789ed79de4aa120d8f"}
Oct 30 00:58:28.046394 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:28.046125    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s" podStartSLOduration=2.046110379 podStartE2EDuration="2.046110379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:28.043550441 +0000 UTC m=+949.591240938" watchObservedRunningTime="2026-10-30 00:58:28.046110379 +0000 UTC m=+949.593800875"
Oct 30 00:58:29.017397 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:29.017360    5104 generic.go:356] "Generic (PLEG): container finished" podID="ef1482d1-eb55-4458-89ca-b766d00d313a" containerID="08919208f41897f733481de3b620133c67a16796292288524135cb07b2855a5f" exitCode=0
Oct 30 00:58:29.017913 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:29.017493    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s" event={"ID":"ef1482d1-eb55-4458-89ca-b766d00d313a","Type":"ContainerDied","Data":"08919208f41897f733481de3b620133c67a16796292288524135cb07b2855a5f"}
Oct 30 00:58:30.205991 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.205963    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:30.328525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.328407    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1482d1-eb55-4458-89ca-b766d00d313a-kube-api-access-z5646\" (UniqueName: \"kubernetes.io/projected/ef1482d1-eb55-4458-89ca-b766d00d313a-kube-api-access-z5646\") pod \"ef1482d1-eb55-4458-89ca-b766d00d313a\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") "
Oct 30 00:58:30.328525 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.328522    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-xks-ca\") pod \"ef1482d1-eb55-4458-89ca-b766d00d313a\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") "
Oct 30 00:58:30.328787 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.328566    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-openshift-ca\") pod \"ef1482d1-eb55-4458-89ca-b766d00d313a\" (UID: \"ef1482d1-eb55-4458-89ca-b766d00d313a\") "
Oct 30 00:58:30.329285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.328965    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-openshift-ca" pod "ef1482d1-eb55-4458-89ca-b766d00d313a" (UID: "ef1482d1-eb55-4458-89ca-b766d00d313a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:30.329285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.329043    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:30.330845 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.330809    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1482d1-eb55-4458-89ca-b766d00d313a-kube-api-access-z5646" pod "ef1482d1-eb55-4458-89ca-b766d00d313a" (UID: "ef1482d1-eb55-4458-89ca-b766d00d313a"). InnerVolumeSpecName "kube-api-access-z5646". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:30.331099 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.331071    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-xks-ca" pod "ef1482d1-eb55-4458-89ca-b766d00d313a" (UID: "ef1482d1-eb55-4458-89ca-b766d00d313a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:30.440481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.431006    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1482d1-eb55-4458-89ca-b766d00d313a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:30.440481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:30.431051    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5646\" (UniqueName: \"kubernetes.io/projected/ef1482d1-eb55-4458-89ca-b766d00d313a-kube-api-access-z5646\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:31.033319 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.033283    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s" event={"ID":"ef1482d1-eb55-4458-89ca-b766d00d313a","Type":"ContainerDied","Data":"03cf21258a4c5ee75b33f74266f5d99a8a15133b3aaaf3789ed79de4aa120d8f"}
Oct 30 00:58:31.033319 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.033305    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"
Oct 30 00:58:31.033570 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.033332    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03cf21258a4c5ee75b33f74266f5d99a8a15133b3aaaf3789ed79de4aa120d8f"
Oct 30 00:58:31.353364 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.353268    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"]
Oct 30 00:58:31.354250 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.354226    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ef1482d1-eb55-4458-89ca-b766d00d313a" containerName="revoke-keys"
Oct 30 00:58:31.354250 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.354250    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1482d1-eb55-4458-89ca-b766d00d313a" containerName="revoke-keys"
Oct 30 00:58:31.354422 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.354280    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c2122562-9974-47ec-8143-039fdb5cb303" containerName="revoke-keys"
Oct 30 00:58:31.354422 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.354289    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2122562-9974-47ec-8143-039fdb5cb303" containerName="revoke-keys"
Oct 30 00:58:31.354580 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.354437    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1482d1-eb55-4458-89ca-b766d00d313a" containerName="revoke-keys"
Oct 30 00:58:31.354580 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.354489    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2122562-9974-47ec-8143-039fdb5cb303" containerName="revoke-keys"
Oct 30 00:58:31.358946 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.358925    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.373993 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.373941    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"]
Oct 30 00:58:31.380558 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.380521    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"distinct-w2-fd3281-kserve-self-signed-certs\""
Oct 30 00:58:31.550483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.547174    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tmp-dir\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.550483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.547252    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-dshm\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.550483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.547277    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tls-certs\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.550483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.547300    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5b2\" (UniqueName: \"kubernetes.io/projected/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-kube-api-access-9c5b2\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.550483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.547417    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-model-cache\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.550483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.547680    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-home\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.648493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648379    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tmp-dir\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.648493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648435    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-dshm\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.648720 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648589    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tls-certs\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.648720 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648621    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5b2\" (UniqueName: \"kubernetes.io/projected/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-kube-api-access-9c5b2\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.648822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648728    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-model-cache\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.648907 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648883    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tmp-dir\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.648976 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648928    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-home\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.649034 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.648985    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-model-cache\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.649213 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.649189    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-home\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.650882 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.650859    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-dshm\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.651198 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.651179    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tls-certs\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.667618 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.667588    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5b2\" (UniqueName: \"kubernetes.io/projected/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-kube-api-access-9c5b2\") pod \"distinct-w2-fd3281-kserve-7b4f5d499-8kzth\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.676551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.676521    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:31.980436 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:31.980407    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"]
Oct 30 00:58:31.981758 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:31.981727    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bc83b57_7ac2_4457_8b3c_0c844b77bb8f.slice/crio-d031f708e312cb34f2333e65d78ae898621d1758a971de0318af47cc6f2828e0 WatchSource:0}: Error finding container d031f708e312cb34f2333e65d78ae898621d1758a971de0318af47cc6f2828e0: Status 404 returned error can't find the container with id d031f708e312cb34f2333e65d78ae898621d1758a971de0318af47cc6f2828e0
Oct 30 00:58:32.042803 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:32.042769    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" event={"ID":"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f","Type":"ContainerStarted","Data":"d031f708e312cb34f2333e65d78ae898621d1758a971de0318af47cc6f2828e0"}
Oct 30 00:58:33.051540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:33.051496    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" event={"ID":"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f","Type":"ContainerStarted","Data":"d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6"}
Oct 30 00:58:33.094309 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:33.094252    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podStartSLOduration=2.094235748 podStartE2EDuration="2.094235748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:33.09155623 +0000 UTC m=+954.639246725" watchObservedRunningTime="2026-10-30 00:58:33.094235748 +0000 UTC m=+954.641926243"
Oct 30 00:58:33.676722 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:33.676682    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:36.285590 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.285552    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"]
Oct 30 00:58:36.290176 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.290149    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.337489 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.335852    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"]
Oct 30 00:58:36.422083 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422047    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422083 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422082    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bc5\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-kube-api-access-b5bc5\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422111    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422225    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422285    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422328 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422327    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7221b2-0f22-4fe0-8db3-87fba47674af-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422399    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422436    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.422564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.422530    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524063 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524020    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524227 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524173    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524227 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524200    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bc5\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-kube-api-access-b5bc5\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524299 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524245    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524387 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524363    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524526    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524546    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524695 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524570    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7221b2-0f22-4fe0-8db3-87fba47674af-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524695 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524642    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524695 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524653    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524695 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524673    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524858 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524691    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.524962 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.524942    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.525391 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.525364    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7221b2-0f22-4fe0-8db3-87fba47674af-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.527069 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.527045    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.527170 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.527075    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.542652 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.542588    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.542821 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.542799    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bc5\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-kube-api-access-b5bc5\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.610040 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.609999    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:36.829582 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.829505    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"]
Oct 30 00:58:36.835855 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.835830    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:36.849289 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.849262    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:36.850521 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.849564    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:58:36.850521 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.850201    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"]
Oct 30 00:58:36.857164 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.857140    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:58:36.857164 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.857140    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:58:36.931127 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.931098    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"]
Oct 30 00:58:36.931935 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.931912    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:36.931993 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.931953    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:36.932124 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.932109    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/f3fc3af6-5316-49a3-9165-dd46f86ac75a-kube-api-access-n685q\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:36.932495 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:36.932451    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7221b2_0f22_4fe0_8db3_87fba47674af.slice/crio-7db1c3cd45e01f49f66539f46d9e690211a8d0948b07bbf1333510897ff26cbd WatchSource:0}: Error finding container 7db1c3cd45e01f49f66539f46d9e690211a8d0948b07bbf1333510897ff26cbd: Status 404 returned error can't find the container with id 7db1c3cd45e01f49f66539f46d9e690211a8d0948b07bbf1333510897ff26cbd
Oct 30 00:58:36.935751 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.935714    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:36.935830 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.935793    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:36.935879 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:36.935837    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:37.033322 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.033283    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:37.033583 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.033340    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:37.033583 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.033494    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/f3fc3af6-5316-49a3-9165-dd46f86ac75a-kube-api-access-n685q\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:37.034282 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.034076    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:37.036023 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.035997    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:37.063327 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.063289    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/f3fc3af6-5316-49a3-9165-dd46f86ac75a-kube-api-access-n685q\") pod \"maas-api-revoke-sub-29a9164c-t9vbh\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:37.085257 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.085150    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" event={"ID":"0c7221b2-0f22-4fe0-8db3-87fba47674af","Type":"ContainerStarted","Data":"4af27eadd7e3c4ce1f9ec3a8ff54a30ca9ab22add19f56cf200279cd64dae889"}
Oct 30 00:58:37.085257 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.085250    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" event={"ID":"0c7221b2-0f22-4fe0-8db3-87fba47674af","Type":"ContainerStarted","Data":"7db1c3cd45e01f49f66539f46d9e690211a8d0948b07bbf1333510897ff26cbd"}
Oct 30 00:58:37.127651 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.127596    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podStartSLOduration=1.127576186 podStartE2EDuration="1.127576186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:37.12362667 +0000 UTC m=+958.671317166" watchObservedRunningTime="2026-10-30 00:58:37.127576186 +0000 UTC m=+958.675266664"
Oct 30 00:58:37.151018 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.150989    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:37.465367 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.465332    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"]
Oct 30 00:58:37.471551 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:37.471449    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fc3af6_5316_49a3_9165_dd46f86ac75a.slice/crio-e05d799a8dd076115697fd797824142155f9136f81388cc1a5d2c2a8c3d3b7bf WatchSource:0}: Error finding container e05d799a8dd076115697fd797824142155f9136f81388cc1a5d2c2a8c3d3b7bf: Status 404 returned error can't find the container with id e05d799a8dd076115697fd797824142155f9136f81388cc1a5d2c2a8c3d3b7bf
Oct 30 00:58:37.610733 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:37.610697    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:38.116554 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:38.116502    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh" event={"ID":"f3fc3af6-5316-49a3-9165-dd46f86ac75a","Type":"ContainerStarted","Data":"d7349bd83b0233004d03bf22569d952d32e08ffa06b55a6b535690140bcfe191"}
Oct 30 00:58:38.116766 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:38.116559    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh" event={"ID":"f3fc3af6-5316-49a3-9165-dd46f86ac75a","Type":"ContainerStarted","Data":"e05d799a8dd076115697fd797824142155f9136f81388cc1a5d2c2a8c3d3b7bf"}
Oct 30 00:58:38.646636 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:38.634671    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:58:38.646636 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:38.634770    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:58:39.185479 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:58:39.184657    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:58:39.200479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:39.187771    5104 generic.go:356] "Generic (PLEG): container finished" podID="f3fc3af6-5316-49a3-9165-dd46f86ac75a" containerID="d7349bd83b0233004d03bf22569d952d32e08ffa06b55a6b535690140bcfe191" exitCode=0
Oct 30 00:58:39.200479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:39.189039    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh" event={"ID":"f3fc3af6-5316-49a3-9165-dd46f86ac75a","Type":"ContainerDied","Data":"d7349bd83b0233004d03bf22569d952d32e08ffa06b55a6b535690140bcfe191"}
Oct 30 00:58:39.612102 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:39.611567    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:58:39.612102 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:39.611638    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:58:40.441972 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.441943    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:40.610965 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.610870    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:58:40.611241 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.611207    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:58:40.621945 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.619899    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3fc3af6-5316-49a3-9165-dd46f86ac75a-kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/f3fc3af6-5316-49a3-9165-dd46f86ac75a-kube-api-access-n685q\") pod \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") "
Oct 30 00:58:40.621945 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.619982    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-xks-ca\") pod \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") "
Oct 30 00:58:40.621945 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.620091    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-openshift-ca\") pod \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\" (UID: \"f3fc3af6-5316-49a3-9165-dd46f86ac75a\") "
Oct 30 00:58:40.623543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.622364    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-openshift-ca" pod "f3fc3af6-5316-49a3-9165-dd46f86ac75a" (UID: "f3fc3af6-5316-49a3-9165-dd46f86ac75a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:40.631957 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.631792    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-xks-ca" pod "f3fc3af6-5316-49a3-9165-dd46f86ac75a" (UID: "f3fc3af6-5316-49a3-9165-dd46f86ac75a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:40.632278 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.632255    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fc3af6-5316-49a3-9165-dd46f86ac75a-kube-api-access-n685q" pod "f3fc3af6-5316-49a3-9165-dd46f86ac75a" (UID: "f3fc3af6-5316-49a3-9165-dd46f86ac75a"). InnerVolumeSpecName "kube-api-access-n685q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:40.724092 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.723642    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.724092 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.723686    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/f3fc3af6-5316-49a3-9165-dd46f86ac75a-kube-api-access-n685q\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.724092 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:40.723703    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fc3af6-5316-49a3-9165-dd46f86ac75a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:58:41.219413 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:41.219363    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh" event={"ID":"f3fc3af6-5316-49a3-9165-dd46f86ac75a","Type":"ContainerDied","Data":"e05d799a8dd076115697fd797824142155f9136f81388cc1a5d2c2a8c3d3b7bf"}
Oct 30 00:58:41.219877 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:41.219560    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"
Oct 30 00:58:41.220431 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:41.219725    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05d799a8dd076115697fd797824142155f9136f81388cc1a5d2c2a8c3d3b7bf"
Oct 30 00:58:41.611069 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:41.610967    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:58:41.611069 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:41.611031    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:58:41.693480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:41.688849    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:41.716281 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:41.716247    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:42.238325 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:42.238292    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:58:42.611526 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:42.611087    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:58:42.611526 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:42.611146    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:58:43.610529 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:43.610490    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:58:43.610718 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:43.610559    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:58:44.742537 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:44.742502    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:44.743963 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:44.743932    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:44.745193 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:44.745155    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 00:58:51.244558 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.244521    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"]
Oct 30 00:58:51.246353 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.246312    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f3fc3af6-5316-49a3-9165-dd46f86ac75a" containerName="revoke-keys"
Oct 30 00:58:51.246550 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.246530    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3fc3af6-5316-49a3-9165-dd46f86ac75a" containerName="revoke-keys"
Oct 30 00:58:51.246826 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.246812    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3fc3af6-5316-49a3-9165-dd46f86ac75a" containerName="revoke-keys"
Oct 30 00:58:51.267043 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.267012    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"]
Oct 30 00:58:51.267334 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.267311    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361217 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361175    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361417 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361279    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361417 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361329    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361417 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361348    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361417 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361630 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361506    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjsr\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-kube-api-access-vfjsr\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361630 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361536    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361671    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d667e298-1030-4b99-999f-85b8f33bd4c9-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.361714 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.361702    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.472566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.470533    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d667e298-1030-4b99-999f-85b8f33bd4c9-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.472566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.470630    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.472566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.470667    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.472566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.470728    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.472566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.470763    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.472566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.470806    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.473019 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.472695    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d667e298-1030-4b99-999f-85b8f33bd4c9-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.473019 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.472890    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.487269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.486936    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.487269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.487025    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.487269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.487134    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.487269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.487228    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjsr\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-kube-api-access-vfjsr\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.487269 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.487269    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.488608 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.488583    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.493230 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.493201    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.506782 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.502864    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.506782 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.503049    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.508157 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.508133    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjsr\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-kube-api-access-vfjsr\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.583170 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.583135    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:51.902372 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.902344    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"]
Oct 30 00:58:51.902890 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:58:51.902864    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd667e298_1030_4b99_999f_85b8f33bd4c9.slice/crio-9dbdd73e863f7d29347b0e16be749df42c3b4a2f5be7d1b2cc26630bd600d9cf WatchSource:0}: Error finding container 9dbdd73e863f7d29347b0e16be749df42c3b4a2f5be7d1b2cc26630bd600d9cf: Status 404 returned error can't find the container with id 9dbdd73e863f7d29347b0e16be749df42c3b4a2f5be7d1b2cc26630bd600d9cf
Oct 30 00:58:51.906322 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.906279    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:51.906410 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.906375    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:51.906450 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:51.906421    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:58:52.312027 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:52.311995    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" event={"ID":"d667e298-1030-4b99-999f-85b8f33bd4c9","Type":"ContainerStarted","Data":"a46dd6e396dec1a7c0b99391dcebd4e805f64f1c8b58ed461890e919494a0a56"}
Oct 30 00:58:52.312630 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:52.312058    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" event={"ID":"d667e298-1030-4b99-999f-85b8f33bd4c9","Type":"ContainerStarted","Data":"9dbdd73e863f7d29347b0e16be749df42c3b4a2f5be7d1b2cc26630bd600d9cf"}
Oct 30 00:58:52.356068 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:52.356010    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podStartSLOduration=1.355991505 podStartE2EDuration="1.355991505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:52.352874656 +0000 UTC m=+973.900565153" watchObservedRunningTime="2026-10-30 00:58:52.355991505 +0000 UTC m=+973.903682002"
Oct 30 00:58:52.583960 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:52.583869    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:52.585602 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:52.585575    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:58:52.585727 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:52.585634    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:58:53.150477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:53.150124    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:58:53.584481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:53.583947    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:58:53.584481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:53.584016    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:58:54.589487 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:54.588656    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.589487 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:54.588722    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:58:55.586873 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:55.586831    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:58:55.587209 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:55.587168    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:58:56.583972 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:56.583935    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:58:56.584596 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:56.584564    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:58:57.596477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:57.587191    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:58:57.596477 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:57.587257    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:58:58.584298 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:58.584259    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:58:58.584450 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:58.584328    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 00:58:59.710413 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:59.710378    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:59.711332 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:59.711302    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:58:59.711997 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:58:59.711966    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 00:59:04.443560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.443525    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"]
Oct 30 00:59:04.463670 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.463630    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.468946 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.468910    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"]
Oct 30 00:59:04.481614 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.481586    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"distinct-w6-c4847c-kserve-self-signed-certs\""
Oct 30 00:59:04.499827 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.499796    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-model-cache\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.499991 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.499858    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svz9x\" (UniqueName: \"kubernetes.io/projected/d5188ca8-b09c-476e-97e7-329244af24f3-kube-api-access-svz9x\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.499991 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.499962    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-tmp-dir\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.500086 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.500051    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5188ca8-b09c-476e-97e7-329244af24f3-tls-certs\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.500132 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.500089    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-dshm\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.500188 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.500175    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-home\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.600847 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.600814    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-dshm\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601050 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.600919    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-home\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601124 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.601066    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-model-cache\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601191 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.601166    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svz9x\" (UniqueName: \"kubernetes.io/projected/d5188ca8-b09c-476e-97e7-329244af24f3-kube-api-access-svz9x\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601250 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.601214    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-tmp-dir\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601308 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.601253    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5188ca8-b09c-476e-97e7-329244af24f3-tls-certs\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601749 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.601719    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-home\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601880 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.601760    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-model-cache\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.601880 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.601777    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-tmp-dir\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.603781 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.603763    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-dshm\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.604150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.604130    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5188ca8-b09c-476e-97e7-329244af24f3-tls-certs\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.619660 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.619633    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svz9x\" (UniqueName: \"kubernetes.io/projected/d5188ca8-b09c-476e-97e7-329244af24f3-kube-api-access-svz9x\") pod \"distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:04.794036 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:04.793945    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:05.096258 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:05.096226    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"]
Oct 30 00:59:05.098242 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:59:05.098217    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5188ca8_b09c_476e_97e7_329244af24f3.slice/crio-5de2df926fa4ec2abc42d761b5cb6a77bb81f720fc1db26551a30227ec9faeab WatchSource:0}: Error finding container 5de2df926fa4ec2abc42d761b5cb6a77bb81f720fc1db26551a30227ec9faeab: Status 404 returned error can't find the container with id 5de2df926fa4ec2abc42d761b5cb6a77bb81f720fc1db26551a30227ec9faeab
Oct 30 00:59:05.493934 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:05.493264    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" event={"ID":"d5188ca8-b09c-476e-97e7-329244af24f3","Type":"ContainerStarted","Data":"882abdf0000f6914c7c591e5a9142bfcfe75b3163002c5f166653890c0019517"}
Oct 30 00:59:05.493934 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:05.493322    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" event={"ID":"d5188ca8-b09c-476e-97e7-329244af24f3","Type":"ContainerStarted","Data":"5de2df926fa4ec2abc42d761b5cb6a77bb81f720fc1db26551a30227ec9faeab"}
Oct 30 00:59:05.534395 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:05.534344    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podStartSLOduration=1.534328626 podStartE2EDuration="1.534328626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:05.529512681 +0000 UTC m=+987.077203173" watchObservedRunningTime="2026-10-30 00:59:05.534328626 +0000 UTC m=+987.082019122"
Oct 30 00:59:05.794588 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:05.794499    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:09.103302 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:59:09.103269    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:59:13.720606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:13.720564    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:59:14.794486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:14.794427    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:14.833485 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:14.830673    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:15.670059 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:15.670021    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:17.160346 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.160309    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"]
Oct 30 00:59:17.165590 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.165571    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.167262 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.167240    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgbt\" (UniqueName: \"kubernetes.io/projected/356bfb70-27c4-4904-974c-9b3230e8f643-kube-api-access-rxgbt\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.167369 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.167312    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.167495 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.167469    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.181685 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.181639    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"]
Oct 30 00:59:17.188748 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.188714    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:59:17.188748 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.188712    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:17.188961 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.188781    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:59:17.190394 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.190368    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:59:17.268959 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.268925    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgbt\" (UniqueName: \"kubernetes.io/projected/356bfb70-27c4-4904-974c-9b3230e8f643-kube-api-access-rxgbt\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.269145 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.269006    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.269145 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.269058    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.269619 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.269597    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.271432 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.271411    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.289009 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.288982    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgbt\" (UniqueName: \"kubernetes.io/projected/356bfb70-27c4-4904-974c-9b3230e8f643-kube-api-access-rxgbt\") pod \"maas-api-revoke-sub-8a288023-2kzl6\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.481448 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.481404    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:17.854547 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:17.854370    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"]
Oct 30 00:59:17.871378 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:59:17.871348    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356bfb70_27c4_4904_974c_9b3230e8f643.slice/crio-8d436eae477799cc7ed4273855756aa0ce23fac59e495fa515ba2e513ad55900 WatchSource:0}: Error finding container 8d436eae477799cc7ed4273855756aa0ce23fac59e495fa515ba2e513ad55900: Status 404 returned error can't find the container with id 8d436eae477799cc7ed4273855756aa0ce23fac59e495fa515ba2e513ad55900
Oct 30 00:59:18.676258 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:18.676224    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6" event={"ID":"356bfb70-27c4-4904-974c-9b3230e8f643","Type":"ContainerStarted","Data":"75b012db3d60c0b16d881ed661e099fcc91647b1fc1821bdd932ffc765c57cf6"}
Oct 30 00:59:18.676709 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:18.676281    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6" event={"ID":"356bfb70-27c4-4904-974c-9b3230e8f643","Type":"ContainerStarted","Data":"8d436eae477799cc7ed4273855756aa0ce23fac59e495fa515ba2e513ad55900"}
Oct 30 00:59:18.714806 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:18.714738    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6" podStartSLOduration=1.714722158 podStartE2EDuration="1.714722158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:18.711224292 +0000 UTC m=+1000.258914789" watchObservedRunningTime="2026-10-30 00:59:18.714722158 +0000 UTC m=+1000.262412653"
Oct 30 00:59:19.684957 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:19.684921    5104 generic.go:356] "Generic (PLEG): container finished" podID="356bfb70-27c4-4904-974c-9b3230e8f643" containerID="75b012db3d60c0b16d881ed661e099fcc91647b1fc1821bdd932ffc765c57cf6" exitCode=0
Oct 30 00:59:19.685358 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:19.685006    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6" event={"ID":"356bfb70-27c4-4904-974c-9b3230e8f643","Type":"ContainerDied","Data":"75b012db3d60c0b16d881ed661e099fcc91647b1fc1821bdd932ffc765c57cf6"}
Oct 30 00:59:20.834864 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.834827    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"]
Oct 30 00:59:20.840921 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.840896    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.854772 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.854740    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c104b8-openshift-default-dockercfg-m9r8p\""
Oct 30 00:59:20.872177 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.872142    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"]
Oct 30 00:59:20.876665 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.876640    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:20.927917 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.927876    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-xks-ca\") pod \"356bfb70-27c4-4904-974c-9b3230e8f643\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") "
Oct 30 00:59:20.928104 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.927967    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356bfb70-27c4-4904-974c-9b3230e8f643-kube-api-access-rxgbt\" (UniqueName: \"kubernetes.io/projected/356bfb70-27c4-4904-974c-9b3230e8f643-kube-api-access-rxgbt\") pod \"356bfb70-27c4-4904-974c-9b3230e8f643\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") "
Oct 30 00:59:20.928178 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.928112    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-openshift-ca\") pod \"356bfb70-27c4-4904-974c-9b3230e8f643\" (UID: \"356bfb70-27c4-4904-974c-9b3230e8f643\") "
Oct 30 00:59:20.928422 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.928393    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.928685 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.928664    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.928846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.928823    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.928952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.928479    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-openshift-ca" pod "356bfb70-27c4-4904-974c-9b3230e8f643" (UID: "356bfb70-27c4-4904-974c-9b3230e8f643"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:20.928952 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.928885    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.929042 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.928992    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.929131 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.929112    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.929584 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.929225    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6rq\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-kube-api-access-7h6rq\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.929584 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.929331    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.929584 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.929386    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:20.930564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.930543    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:20.930701 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.930590    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-xks-ca" pod "356bfb70-27c4-4904-974c-9b3230e8f643" (UID: "356bfb70-27c4-4904-974c-9b3230e8f643"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:20.930789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:20.930729    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356bfb70-27c4-4904-974c-9b3230e8f643-kube-api-access-rxgbt" pod "356bfb70-27c4-4904-974c-9b3230e8f643" (UID: "356bfb70-27c4-4904-974c-9b3230e8f643"). InnerVolumeSpecName "kube-api-access-rxgbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:21.031645 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031542    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6rq\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-kube-api-access-7h6rq\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.031876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031647    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.031876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031677    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.031876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031753    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.031876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031799    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.031876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031828    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.031876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031874    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.032189 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.031928    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.032189 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032001    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.032189 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032095    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356bfb70-27c4-4904-974c-9b3230e8f643-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:21.032189 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032113    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgbt\" (UniqueName: \"kubernetes.io/projected/356bfb70-27c4-4904-974c-9b3230e8f643-kube-api-access-rxgbt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:21.032189 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032137    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.032476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032200    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.032476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032370    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.032579 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032445    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.032703 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.032682    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.034145 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.034126    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.034294 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.034272    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.050498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.050467    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.050649 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.050567    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6rq\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-kube-api-access-7h6rq\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.194469 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.194428    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:21.562254 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.562222    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"]
Oct 30 00:59:21.562812 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:59:21.562787    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cfbc0c_77b6_4a20_97b5_e53787626b3a.slice/crio-5b3403b90b9bc9cebc926cfbff9d3232c5aeb5e65fdff8c307312180bd5f3dc9 WatchSource:0}: Error finding container 5b3403b90b9bc9cebc926cfbff9d3232c5aeb5e65fdff8c307312180bd5f3dc9: Status 404 returned error can't find the container with id 5b3403b90b9bc9cebc926cfbff9d3232c5aeb5e65fdff8c307312180bd5f3dc9
Oct 30 00:59:21.565239 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.565207    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:21.565311 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.565273    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:21.565311 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.565301    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:21.708476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.706113    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"
Oct 30 00:59:21.708476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.706848    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6" event={"ID":"356bfb70-27c4-4904-974c-9b3230e8f643","Type":"ContainerDied","Data":"8d436eae477799cc7ed4273855756aa0ce23fac59e495fa515ba2e513ad55900"}
Oct 30 00:59:21.708476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.706965    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d436eae477799cc7ed4273855756aa0ce23fac59e495fa515ba2e513ad55900"
Oct 30 00:59:21.716501 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.714711    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" event={"ID":"79cfbc0c-77b6-4a20-97b5-e53787626b3a","Type":"ContainerStarted","Data":"4798382ae5e62f842ca4becc398b9deff0d4c9094c6c8f9d41099aacd363605b"}
Oct 30 00:59:21.716501 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.714769    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" event={"ID":"79cfbc0c-77b6-4a20-97b5-e53787626b3a","Type":"ContainerStarted","Data":"5b3403b90b9bc9cebc926cfbff9d3232c5aeb5e65fdff8c307312180bd5f3dc9"}
Oct 30 00:59:21.755803 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:21.755738    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podStartSLOduration=1.755719239 podStartE2EDuration="1.755719239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:21.752451117 +0000 UTC m=+1003.300141613" watchObservedRunningTime="2026-10-30 00:59:21.755719239 +0000 UTC m=+1003.303409739"
Oct 30 00:59:22.195342 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:22.195303    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:22.197226 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:22.197194    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:59:22.197365 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:22.197262    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:59:23.199489 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:23.198902    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:59:23.199489 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:23.198969    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:59:24.199481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:24.195865    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:59:24.199481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:24.195931    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:59:25.200481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:25.195858    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:59:25.200481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:25.195922    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:59:26.194889 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:26.194848    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:59:26.195232 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:26.195191    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:59:26.946479 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:26.945773    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"]
Oct 30 00:59:26.947135 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:26.947096    5104 cpu_manager.go:414] "containerMap: removing container" podUID="356bfb70-27c4-4904-974c-9b3230e8f643" containerName="revoke-keys"
Oct 30 00:59:26.947135 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:26.947119    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356bfb70-27c4-4904-974c-9b3230e8f643" containerName="revoke-keys"
Oct 30 00:59:26.947339 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:26.947238    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="356bfb70-27c4-4904-974c-9b3230e8f643" containerName="revoke-keys"
Oct 30 00:59:26.951480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:26.950775    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.006486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.000833    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"kube-root-ca.crt\""
Oct 30 00:59:27.006486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.001623    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"default-dockercfg-2kgzn\""
Oct 30 00:59:27.006486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.001735    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"distinct-2-w1-c104b8-kserve-self-signed-certs\""
Oct 30 00:59:27.006486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.001810    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"openshift-service-ca.crt\""
Oct 30 00:59:27.012287 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.012255    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"]
Oct 30 00:59:27.060864 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.060828    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-dshm\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.061076 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.060883    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-model-cache\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.061076 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.061009    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aad504-0d68-4aa8-897a-6f443646e3c3-tls-certs\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.061076 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.061054    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-tmp-dir\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.061285 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.061180    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-home\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.061351 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.061306    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wph2s\" (UniqueName: \"kubernetes.io/projected/f7aad504-0d68-4aa8-897a-6f443646e3c3-kube-api-access-wph2s\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.170385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-home\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.170533    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wph2s\" (UniqueName: \"kubernetes.io/projected/f7aad504-0d68-4aa8-897a-6f443646e3c3-kube-api-access-wph2s\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.170658    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-dshm\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.170705    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-model-cache\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.170744    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aad504-0d68-4aa8-897a-6f443646e3c3-tls-certs\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.170787    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-tmp-dir\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.171369    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-tmp-dir\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.174481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.171707    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-home\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.175502 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.175448    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-model-cache\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.201276 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.201197    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:59:27.201560 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.201525    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:59:27.243721 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.204231    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-dshm\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.243721 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.213225    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aad504-0d68-4aa8-897a-6f443646e3c3-tls-certs\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.243721 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.225325    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wph2s\" (UniqueName: \"kubernetes.io/projected/f7aad504-0d68-4aa8-897a-6f443646e3c3-kube-api-access-wph2s\") pod \"distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.271483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.270702    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:27.741626 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.741586    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"]
Oct 30 00:59:27.765120 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:27.765035    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" event={"ID":"f7aad504-0d68-4aa8-897a-6f443646e3c3","Type":"ContainerStarted","Data":"cf910493d8736b039b3f7a73181a773bebb30630f8f667849bd94f92a9811c05"}
Oct 30 00:59:28.195607 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.195559    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 00:59:28.195979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.195643    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 00:59:28.779352 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.779314    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" event={"ID":"f7aad504-0d68-4aa8-897a-6f443646e3c3","Type":"ContainerStarted","Data":"c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968"}
Oct 30 00:59:28.781622 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.781594    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"]
Oct 30 00:59:28.786319 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.786297    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:28.800557 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.800520    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"]
Oct 30 00:59:28.805011 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.804981    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:59:28.805438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.805161    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:59:28.805438 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.805379    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:59:28.805806 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.805570    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:28.833547 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.833484    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podStartSLOduration=2.833446193 podStartE2EDuration="2.833446193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:28.826857981 +0000 UTC m=+1010.374548474" watchObservedRunningTime="2026-10-30 00:59:28.833446193 +0000 UTC m=+1010.381136691"
Oct 30 00:59:28.878931 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.877711    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:28.878931 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.877857    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtwl\" (UniqueName: \"kubernetes.io/projected/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-kube-api-access-7jtwl\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:28.878931 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.877916    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:28.993244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.992329    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:28.993244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.992511    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtwl\" (UniqueName: \"kubernetes.io/projected/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-kube-api-access-7jtwl\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:28.993244 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.992579    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:28.994533 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:28.994501    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:29.008978 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.008945    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:29.025480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.024101    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtwl\" (UniqueName: \"kubernetes.io/projected/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-kube-api-access-7jtwl\") pod \"maas-api-revoke-sub-1e53bb6b-4wdlv\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:29.116363 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.116251    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:29.201061 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.200628    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:29.202869 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.201957    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:29.202991 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.202969    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 00:59:29.274518 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.272965    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:29.452847 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.452818    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"]
Oct 30 00:59:29.453661 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:59:29.453622    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e07e027_2a84_4dd9_99b7_c21acea9c8f2.slice/crio-0322e64d3c02bd6fb13c2476febefe2b817a1925cd9a0a06bc335242de230975 WatchSource:0}: Error finding container 0322e64d3c02bd6fb13c2476febefe2b817a1925cd9a0a06bc335242de230975: Status 404 returned error can't find the container with id 0322e64d3c02bd6fb13c2476febefe2b817a1925cd9a0a06bc335242de230975
Oct 30 00:59:29.789897 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:29.789870    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv" event={"ID":"3e07e027-2a84-4dd9-99b7-c21acea9c8f2","Type":"ContainerStarted","Data":"0322e64d3c02bd6fb13c2476febefe2b817a1925cd9a0a06bc335242de230975"}
Oct 30 00:59:30.798481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:30.798364    5104 generic.go:356] "Generic (PLEG): container finished" podID="3e07e027-2a84-4dd9-99b7-c21acea9c8f2" containerID="4b40bde495243869c7b7cc5351bdbad37975ebaeea1116a4b541c160e8b7f20c" exitCode=0
Oct 30 00:59:30.798912 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:30.798497    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv" event={"ID":"3e07e027-2a84-4dd9-99b7-c21acea9c8f2","Type":"ContainerDied","Data":"4b40bde495243869c7b7cc5351bdbad37975ebaeea1116a4b541c160e8b7f20c"}
Oct 30 00:59:31.977691 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:31.977653    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:32.055624 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.055589    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-xks-ca\") pod \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") "
Oct 30 00:59:32.055842 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.055768    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-kube-api-access-7jtwl\" (UniqueName: \"kubernetes.io/projected/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-kube-api-access-7jtwl\") pod \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") "
Oct 30 00:59:32.055912 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.055899    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-openshift-ca\") pod \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\" (UID: \"3e07e027-2a84-4dd9-99b7-c21acea9c8f2\") "
Oct 30 00:59:32.056342 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.056310    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-openshift-ca" pod "3e07e027-2a84-4dd9-99b7-c21acea9c8f2" (UID: "3e07e027-2a84-4dd9-99b7-c21acea9c8f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:32.057162 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.056983    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:32.058074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.058043    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-xks-ca" pod "3e07e027-2a84-4dd9-99b7-c21acea9c8f2" (UID: "3e07e027-2a84-4dd9-99b7-c21acea9c8f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:32.058426 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.058405    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-kube-api-access-7jtwl" pod "3e07e027-2a84-4dd9-99b7-c21acea9c8f2" (UID: "3e07e027-2a84-4dd9-99b7-c21acea9c8f2"). InnerVolumeSpecName "kube-api-access-7jtwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:32.158139 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.158102    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:32.158139 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.158134    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtwl\" (UniqueName: \"kubernetes.io/projected/3e07e027-2a84-4dd9-99b7-c21acea9c8f2-kube-api-access-7jtwl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:32.814574 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.814541    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"
Oct 30 00:59:32.814779 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.814576    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv" event={"ID":"3e07e027-2a84-4dd9-99b7-c21acea9c8f2","Type":"ContainerDied","Data":"0322e64d3c02bd6fb13c2476febefe2b817a1925cd9a0a06bc335242de230975"}
Oct 30 00:59:32.814779 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:32.814627    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0322e64d3c02bd6fb13c2476febefe2b817a1925cd9a0a06bc335242de230975"
Oct 30 00:59:36.182820 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.182772    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"]
Oct 30 00:59:36.183514 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.183495    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3e07e027-2a84-4dd9-99b7-c21acea9c8f2" containerName="revoke-keys"
Oct 30 00:59:36.183514 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.183514    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e07e027-2a84-4dd9-99b7-c21acea9c8f2" containerName="revoke-keys"
Oct 30 00:59:36.183646 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.183636    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e07e027-2a84-4dd9-99b7-c21acea9c8f2" containerName="revoke-keys"
Oct 30 00:59:36.187657 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.187631    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.232524 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.231242    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"]
Oct 30 00:59:36.329802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.329758    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.329991 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.329810    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.329991 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.329914    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.329991 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.329935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.330140 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.330045    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.330140 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.330100    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.330334 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.330274    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.330420 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.330335    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzgx\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-kube-api-access-rrzgx\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.330420 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.330369    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.346433 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.346405    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"]
Oct 30 00:59:36.350802 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.350776    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.373025 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.372980    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"]
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436494    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436559    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436610    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436635    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436720    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436767    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436917    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.436975    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzgx\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-kube-api-access-rrzgx\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.440565 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.437013    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.508540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.494973    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.508540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.495598    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.508540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.495749    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.508540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.495928    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.508540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.496191    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.508906 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.508625    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538439    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538587    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c26b785-d2fc-428b-b7cd-14dd05db4270-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538636    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqwv\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-kube-api-access-fjqwv\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538669    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538693    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538754    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538893    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.538938    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.539005    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.546023    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.558566 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.548297    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzgx\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-kube-api-access-rrzgx\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.560271 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.560228    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.640356 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640321    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c26b785-d2fc-428b-b7cd-14dd05db4270-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.640613 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640379    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwv\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-kube-api-access-fjqwv\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.640613 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640520    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.640613 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640551    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.640613 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640606    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.640876 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640853    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.640929 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640908    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.640992 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.640976    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.641048 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.641010    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.641103 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.641086    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.641212 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.641185    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.641265 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.641196    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c26b785-d2fc-428b-b7cd-14dd05db4270-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.641315 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.641286    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.641412 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.641396    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.642917 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.642890    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.643379 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.643360    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.657959 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.657934    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.658186 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.658165    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqwv\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-kube-api-access-fjqwv\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.667042 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.667009    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:36.803621 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.803299    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:36.989321 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:59:36.989020    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c26b785_d2fc_428b_b7cd_14dd05db4270.slice/crio-3d286636fa934f55c6b0f4f9f56d223ceaf18d0b601ac59f4488334a6bf675ca WatchSource:0}: Error finding container 3d286636fa934f55c6b0f4f9f56d223ceaf18d0b601ac59f4488334a6bf675ca: Status 404 returned error can't find the container with id 3d286636fa934f55c6b0f4f9f56d223ceaf18d0b601ac59f4488334a6bf675ca
Oct 30 00:59:36.989321 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.989219    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"]
Oct 30 00:59:36.993543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.991860    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:36.993543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.991922    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:36.993543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:36.991952    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:37.204907 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.204856    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 00:59:37.206960 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:59:37.206930    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0b5121_f9ca_4203_9a44_ad3980d3f439.slice/crio-1eaa9cb77473566278b837a0ef08307e03e5bc33599a73f30caee37f905d3199 WatchSource:0}: Error finding container 1eaa9cb77473566278b837a0ef08307e03e5bc33599a73f30caee37f905d3199: Status 404 returned error can't find the container with id 1eaa9cb77473566278b837a0ef08307e03e5bc33599a73f30caee37f905d3199
Oct 30 00:59:37.210372 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.210327    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:37.210510 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.210424    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:37.210510 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.210491    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:37.211214 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.211195    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"]
Oct 30 00:59:37.273145 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.273118    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:37.289639 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.289611    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:37.954482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.946471    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" event={"ID":"7f0b5121-f9ca-4203-9a44-ad3980d3f439","Type":"ContainerStarted","Data":"c82557edd652cbc19c56abd0c2fa5027e3d76fc62515d9b4e0c1ad22649d851f"}
Oct 30 00:59:37.954482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.946545    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" event={"ID":"7f0b5121-f9ca-4203-9a44-ad3980d3f439","Type":"ContainerStarted","Data":"1eaa9cb77473566278b837a0ef08307e03e5bc33599a73f30caee37f905d3199"}
Oct 30 00:59:37.963487 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.961443    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" event={"ID":"3c26b785-d2fc-428b-b7cd-14dd05db4270","Type":"ContainerStarted","Data":"ddcd629386378411440037556c0d7d7859804e37a7c916b6a10efbfff7ce1d4a"}
Oct 30 00:59:37.963487 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:37.961528    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" event={"ID":"3c26b785-d2fc-428b-b7cd-14dd05db4270","Type":"ContainerStarted","Data":"3d286636fa934f55c6b0f4f9f56d223ceaf18d0b601ac59f4488334a6bf675ca"}
Oct 30 00:59:38.036653 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.036602    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podStartSLOduration=2.036584707 podStartE2EDuration="2.036584707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:38.013936989 +0000 UTC m=+1019.561627486" watchObservedRunningTime="2026-10-30 00:59:38.036584707 +0000 UTC m=+1019.584275201"
Oct 30 00:59:38.036964 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.036938    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:38.055734 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.055693    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podStartSLOduration=2.055681207 podStartE2EDuration="2.055681207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:38.054414775 +0000 UTC m=+1019.602105270" watchObservedRunningTime="2026-10-30 00:59:38.055681207 +0000 UTC m=+1019.603371703"
Oct 30 00:59:38.687846 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.687806    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 00:59:38.696829 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.696763    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 00:59:38.845480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.826245    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:38.845480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.839970    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:38.855494 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.852065    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:59:38.855494 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:38.852154    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:59:39.231483 ip-10-0-130-169 kubenswrapper[5104]: E1030 00:59:39.230355    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 00:59:39.668534 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:39.667702    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 00:59:39.668534 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:39.667760    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 00:59:39.811483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:39.810907    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:59:39.811483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:39.810987    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:59:40.667515 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:40.667446    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 00:59:40.667680 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:40.667551    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 00:59:40.824483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:40.812906    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:59:40.824483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:40.813009    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:59:41.667700 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:41.667667    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 00:59:41.667890 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:41.667728    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 00:59:41.811482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:41.807696    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:59:41.811482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:41.807771    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:59:42.644371 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.644330    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"]
Oct 30 00:59:42.651067 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.651041    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.657747 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.657720    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"]
Oct 30 00:59:42.666800 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.666769    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:42.667671 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.667646    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 00:59:42.667837 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.667817    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.812750    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdst\" (UniqueName: \"kubernetes.io/projected/e4c0cace-b390-4fe2-962a-f0358df08b41-kube-api-access-ltdst\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.812957    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.812990    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.856148    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.857608    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.857685    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.870647    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.920474    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.920522    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.953538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.920685    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdst\" (UniqueName: \"kubernetes.io/projected/e4c0cace-b390-4fe2-962a-f0358df08b41-kube-api-access-ltdst\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.961519 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.961484    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.964799 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.960614    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:59:42.965376 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.965341    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.969789 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.969746    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdst\" (UniqueName: \"kubernetes.io/projected/e4c0cace-b390-4fe2-962a-f0358df08b41-kube-api-access-ltdst\") pod \"maas-api-revoke-sub-c97b1dfe-l572h\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:42.971657 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:42.971487    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:43.425008 ip-10-0-130-169 kubenswrapper[5104]: W1030 00:59:43.424974    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c0cace_b390_4fe2_962a_f0358df08b41.slice/crio-ef2ec5c3b9363f7a37d65f9ad19307ab68c223a0c21c0388ac881fb57a67d276 WatchSource:0}: Error finding container ef2ec5c3b9363f7a37d65f9ad19307ab68c223a0c21c0388ac881fb57a67d276: Status 404 returned error can't find the container with id ef2ec5c3b9363f7a37d65f9ad19307ab68c223a0c21c0388ac881fb57a67d276
Oct 30 00:59:43.427449 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:43.427416    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"]
Oct 30 00:59:43.667612 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:43.667562    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 00:59:43.668020 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:43.667640    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 00:59:43.804643 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:43.804604    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 00:59:43.804827 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:43.804675    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 00:59:44.129698 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:44.129591    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h" event={"ID":"e4c0cace-b390-4fe2-962a-f0358df08b41","Type":"ContainerStarted","Data":"ef2de4f7048be0ba53d2e4f508c4552816d614fd27c96f6d837e9b1ea2f7303d"}
Oct 30 00:59:44.129698 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:44.129661    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h" event={"ID":"e4c0cace-b390-4fe2-962a-f0358df08b41","Type":"ContainerStarted","Data":"ef2ec5c3b9363f7a37d65f9ad19307ab68c223a0c21c0388ac881fb57a67d276"}
Oct 30 00:59:44.165955 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:44.165893    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h" podStartSLOduration=2.165878882 podStartE2EDuration="2.165878882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:44.165012325 +0000 UTC m=+1025.712702834" watchObservedRunningTime="2026-10-30 00:59:44.165878882 +0000 UTC m=+1025.713569378"
Oct 30 00:59:44.721695 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:44.720742    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:44.722346 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:44.722294    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:44.726979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:44.726951    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 00:59:45.142160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:45.142117    5104 generic.go:356] "Generic (PLEG): container finished" podID="e4c0cace-b390-4fe2-962a-f0358df08b41" containerID="ef2de4f7048be0ba53d2e4f508c4552816d614fd27c96f6d837e9b1ea2f7303d" exitCode=0
Oct 30 00:59:45.142375 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:45.142201    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h" event={"ID":"e4c0cace-b390-4fe2-962a-f0358df08b41","Type":"ContainerDied","Data":"ef2de4f7048be0ba53d2e4f508c4552816d614fd27c96f6d837e9b1ea2f7303d"}
Oct 30 00:59:45.811443 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:45.811324    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:59:45.811987 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:45.811546    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.525482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.524961    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:46.666099 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.666056    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-openshift-ca\") pod \"e4c0cace-b390-4fe2-962a-f0358df08b41\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") "
Oct 30 00:59:46.666294 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.666213    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-xks-ca\") pod \"e4c0cace-b390-4fe2-962a-f0358df08b41\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") "
Oct 30 00:59:46.666294 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.666260    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c0cace-b390-4fe2-962a-f0358df08b41-kube-api-access-ltdst\" (UniqueName: \"kubernetes.io/projected/e4c0cace-b390-4fe2-962a-f0358df08b41-kube-api-access-ltdst\") pod \"e4c0cace-b390-4fe2-962a-f0358df08b41\" (UID: \"e4c0cace-b390-4fe2-962a-f0358df08b41\") "
Oct 30 00:59:46.666564 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.666510    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-openshift-ca" pod "e4c0cace-b390-4fe2-962a-f0358df08b41" (UID: "e4c0cace-b390-4fe2-962a-f0358df08b41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:46.667365 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.667337    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:46.668572 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.668543    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-xks-ca" pod "e4c0cace-b390-4fe2-962a-f0358df08b41" (UID: "e4c0cace-b390-4fe2-962a-f0358df08b41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:46.668673 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.668608    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c0cace-b390-4fe2-962a-f0358df08b41-kube-api-access-ltdst" pod "e4c0cace-b390-4fe2-962a-f0358df08b41" (UID: "e4c0cace-b390-4fe2-962a-f0358df08b41"). InnerVolumeSpecName "kube-api-access-ltdst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:46.768503 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.768390    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c0cace-b390-4fe2-962a-f0358df08b41-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:46.768503 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.768424    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdst\" (UniqueName: \"kubernetes.io/projected/e4c0cace-b390-4fe2-962a-f0358df08b41-kube-api-access-ltdst\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 00:59:46.812154 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.812111    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:59:46.812615 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.812200    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.842618 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.842574    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:46.843195 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.843168    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:46.846641 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:46.846602    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 00:59:47.157739 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:47.157650    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"
Oct 30 00:59:47.157739 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:47.157688    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h" event={"ID":"e4c0cace-b390-4fe2-962a-f0358df08b41","Type":"ContainerDied","Data":"ef2ec5c3b9363f7a37d65f9ad19307ab68c223a0c21c0388ac881fb57a67d276"}
Oct 30 00:59:47.157739 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:47.157735    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef2ec5c3b9363f7a37d65f9ad19307ab68c223a0c21c0388ac881fb57a67d276"
Oct 30 00:59:51.312784 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.312699    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"]
Oct 30 00:59:51.323150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.323024    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e4c0cace-b390-4fe2-962a-f0358df08b41" containerName="revoke-keys"
Oct 30 00:59:51.323150 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.323056    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c0cace-b390-4fe2-962a-f0358df08b41" containerName="revoke-keys"
Oct 30 00:59:51.323393 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.323265    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c0cace-b390-4fe2-962a-f0358df08b41" containerName="revoke-keys"
Oct 30 00:59:51.329210 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.329182    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"]
Oct 30 00:59:51.329372 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.329302    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.336750 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.336715    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"]
Oct 30 00:59:51.336917 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.336814    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"]
Oct 30 00:59:51.336917 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.336833    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.436932 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.436896    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.436932 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.436929    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.437184 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.436950    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.437184 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437113    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437273 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437183    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437310 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437277    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h856j\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-kube-api-access-h856j\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.437366 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437315    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2xc\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-kube-api-access-wt2xc\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437441 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437420    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437528 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437507    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437601 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437584    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437704 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437691    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437764 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437723    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d96c09-fa3c-419a-8036-7189095b73e4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.437807 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437758    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.437887 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437863    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437933 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437911    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e8adc-8578-4335-86db-dd05f44ff6c4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.437993 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.437978    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d96c09-fa3c-419a-8036-7189095b73e4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.438047 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.438032    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.438115 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.438096    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.539605 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539563    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.539822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539628    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.539822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539673    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h856j\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-kube-api-access-h856j\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.539822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539692    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2xc\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-kube-api-access-wt2xc\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.539822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539717    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.539822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539740    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.539822 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539787    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539938    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.539973    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d96c09-fa3c-419a-8036-7189095b73e4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540005    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540034    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e8adc-8578-4335-86db-dd05f44ff6c4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540069    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540099    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d96c09-fa3c-419a-8036-7189095b73e4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540115    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540161    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540222    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540268    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540272    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540341    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540374    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540400    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540508    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.540610 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540605    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.541074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540871    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.541074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.540918    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e8adc-8578-4335-86db-dd05f44ff6c4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.541074 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.541019    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d96c09-fa3c-419a-8036-7189095b73e4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.541326 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.541306    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.542685 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.542660    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.542814 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.542710    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.543089 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.543065    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d96c09-fa3c-419a-8036-7189095b73e4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.543166 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.543149    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.558506 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.558480    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h856j\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-kube-api-access-h856j\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.558980 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.558954    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2xc\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-kube-api-access-wt2xc\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.559063 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.558965    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.559110 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.559100    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.650801 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.650712    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:51.659214 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.659190    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:51.737535 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.737500    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"]
Oct 30 00:59:51.744854 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.744811    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.782556 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.782529    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"]
Oct 30 00:59:51.845796 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.845635    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.845796 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.845716    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/973d2316-df2e-4774-b177-1b799d70f8e9-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.845796 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.845758    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.846105 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.845847    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/973d2316-df2e-4774-b177-1b799d70f8e9-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.846105 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.845893    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.846105 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.845949    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.846105 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.846011    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.846312 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.846130    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.846312 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.846155    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swz7f\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-kube-api-access-swz7f\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.973599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.969943    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.973599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.969995    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swz7f\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-kube-api-access-swz7f\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.973599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.970077    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.973599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.970145    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/973d2316-df2e-4774-b177-1b799d70f8e9-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.973599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.970184    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.973599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.970318    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/973d2316-df2e-4774-b177-1b799d70f8e9-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.973599 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.970373    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.986543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.985583    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.986543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.985686    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.999793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.987591    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.999793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.987848    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:51.999793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.988084    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:52.000274 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.989283    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/973d2316-df2e-4774-b177-1b799d70f8e9-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:52.000514 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:51.999330    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:52.014032 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.013094    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/973d2316-df2e-4774-b177-1b799d70f8e9-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:52.014032 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.013306    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:52.046583 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.046514    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swz7f\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-kube-api-access-swz7f\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:52.067314 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.067282    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"]
Oct 30 00:59:52.111267 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.110151    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:52.111544 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.111517    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:52.111824 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.111801    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:52.112555 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.110577    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:52.112739 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.112718    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:52.112887 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.112868    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:52.116627 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.111176    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:52.116627 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.116415    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"]
Oct 30 00:59:52.240886 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.240784    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" event={"ID":"d00e8adc-8578-4335-86db-dd05f44ff6c4","Type":"ContainerStarted","Data":"fd7d378f06132bfe0b50f93f0e3ccaf06f94849b1be5632b9413f435abe01a78"}
Oct 30 00:59:52.246507 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.245814    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" event={"ID":"14d96c09-fa3c-419a-8036-7189095b73e4","Type":"ContainerStarted","Data":"436348ecca89789af97b7a9a41975bc619e286e36854f194808091229f51e4a8"}
Oct 30 00:59:52.399557 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:52.398728    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:53.527592 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:53.527519    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"]
Oct 30 00:59:53.675656 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:53.673583    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" event={"ID":"d00e8adc-8578-4335-86db-dd05f44ff6c4","Type":"ContainerStarted","Data":"ce05a6d0d3bd11cf03f795e157a1df6ea1b9562270fc7a2923d7827aae7286e6"}
Oct 30 00:59:53.693003 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:53.692963    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" event={"ID":"14d96c09-fa3c-419a-8036-7189095b73e4","Type":"ContainerStarted","Data":"2650eb0e427f4305faf7dfa9ecd3b2d84f2c1509876635546273c560f4d15e86"}
Oct 30 00:59:53.873545 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:53.832273    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:53.873545 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:53.843516    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:53.873545 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:53.843881    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 00:59:54.096307 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.066106    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 00:59:54.421482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.398009    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podStartSLOduration=3.397983623 podStartE2EDuration="3.397983623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:53.930278459 +0000 UTC m=+1035.477968955" watchObservedRunningTime="2026-10-30 00:59:54.397983623 +0000 UTC m=+1035.945674113"
Oct 30 00:59:54.433552 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.424059    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podStartSLOduration=3.42403923 podStartE2EDuration="3.42403923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:54.036158038 +0000 UTC m=+1035.583848535" watchObservedRunningTime="2026-10-30 00:59:54.42403923 +0000 UTC m=+1035.971729728"
Oct 30 00:59:54.475635 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.475585    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:54.476409 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.476372    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:54.634483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.629637    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:54.648426 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.648373    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:54.661520 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.660677    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:59:54.661520 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.660757    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 00:59:54.677335 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.676699    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:59:54.677335 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.676803    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:59:54.701980 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.701948    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 00:59:54.702278 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.702250    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 00:59:54.827723 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:54.827684    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" event={"ID":"973d2316-df2e-4774-b177-1b799d70f8e9","Type":"ContainerStarted","Data":"cbe826444cde700f3e4234dc483f5afd80e16f8840f9bc328dcc256b6fbb95df"}
Oct 30 00:59:55.044868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.044762    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.080606 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.080557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.082131 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.082106    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 00:59:55.165482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.155587    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.285482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.285013    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.314303 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.314188    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.496568 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.491031    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.506481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.503047    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:59:55.549538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.527585    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.549538 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.528275    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:59:55.635160 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.634947    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.649847 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.649713    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.650754 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.646678    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:55.655957 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.655932    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:59:55.733487 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.729594    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:55.842489 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.744971    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:59:55.842489 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.745029    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:59:55.842935 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.842894    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 00:59:56.069543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.852624    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:59:56.069543 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:55.852687    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 00:59:56.157980 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.157930    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:56.344240 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.344137    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:56.344887 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.344853    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 00:59:56.345133 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.345112    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 00:59:56.359832 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.359792    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:59:56.360098 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.360071    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" event={"ID":"973d2316-df2e-4774-b177-1b799d70f8e9","Type":"ContainerStarted","Data":"1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651"}
Oct 30 00:59:56.360281 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.360265    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:59:56.360413 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.360401    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 00:59:56.369551 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.368533    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 00:59:56.386482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.370737    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:59:56.413559 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.413522    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 00:59:56.438484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.426590    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:59:56.438484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.426653    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:59:56.461480 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.445372    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:56.474486 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.472133    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podStartSLOduration=5.472110925 podStartE2EDuration="5.472110925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:56.427114004 +0000 UTC m=+1037.974804510" watchObservedRunningTime="2026-10-30 00:59:56.472110925 +0000 UTC m=+1038.019801422"
Oct 30 00:59:56.553509 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.519118    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:59:56.553509 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.521555    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:59:56.553509 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.532583    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 00:59:56.666377 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.666266    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 00:59:56.718659 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.718619    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:59:56.755604 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.755550    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:59:56.755832 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.720484    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:59:56.755979 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:56.755959    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 00:59:57.249476 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.249414    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:59:57.300484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.293327    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 00:59:57.300484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.293413    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 00:59:57.300484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.293486    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"]
Oct 30 00:59:57.397478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.396253    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:57.397478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.397275    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 00:59:57.467524 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.465937    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:59:57.467524 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.465999    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:59:57.472124 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.472087    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:57.537676 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.537578    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:57.538586 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.537617    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:57.655484 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.645037    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"]
Oct 30 00:59:57.655868 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.650704    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.687539 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.671562    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:57.747702 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.747666    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:59:57.771655 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.771588    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 00:59:57.789532 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.776386    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:59:57.790131 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.790015    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:57.806478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.790268    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:59:57.806478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.790351    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:59:57.806478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.804300    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef07710-514a-4672-b81c-682651eff83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.806478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.804360    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5w46\" (UniqueName: \"kubernetes.io/projected/fef07710-514a-4672-b81c-682651eff83f-kube-api-access-q5w46\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.806478 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.804665    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef07710-514a-4672-b81c-682651eff83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.807059 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.807034    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 00:59:57.933483 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.829296    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:59:57.937626 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.910930    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef07710-514a-4672-b81c-682651eff83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.942493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.938851    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef07710-514a-4672-b81c-682651eff83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.942493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.938899    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w46\" (UniqueName: \"kubernetes.io/projected/fef07710-514a-4672-b81c-682651eff83f-kube-api-access-q5w46\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.981540 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.969643    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef07710-514a-4672-b81c-682651eff83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:57.989187 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:57.989157    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 00:59:58.043006 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.042901    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5w46\" (UniqueName: \"kubernetes.io/projected/fef07710-514a-4672-b81c-682651eff83f-kube-api-access-q5w46\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:58.110482 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.107553    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:58.249493 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.164911    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef07710-514a-4672-b81c-682651eff83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0eca552-cv95n\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:58.292501 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.252034    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:58.300539 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.291772    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:58.330682 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.300830    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:58.410498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.397558    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:58.410498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.399802    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:59:58.410498 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.399854    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:59:58.477508 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.473519    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 00:59:58.529481 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.528840    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 00:59:58.561548 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.556555    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:59:58.561548 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.557721    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:58.578116 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.568655    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 00:59:58.854772 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.782026    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 00:59:58.854772 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.817496    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 30 00:59:58.993677 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.993642    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:59:58.993919 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.993893    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:59:58.994420 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.994394    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:59:58.994594 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:58.994574    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 00:59:59.078162 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.078131    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 00:59:59.273781 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.273749    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 00:59:59.318697 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.311876    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 00:59:59.442575 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.442534    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:59:59.442793 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.442769    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 00:59:59.514709 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.507749    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:59.679539 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.671673    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:59:59.679539 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.673920    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"]
Oct 30 00:59:59.679539 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.676995    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:59:59.679539 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.677046    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 00:59:59.719514 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.694339    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:59:59.753581 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.753539    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 00:59:59.969534 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.960959    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjh6\" (UniqueName: \"kubernetes.io/projected/d3dac020-aeba-427a-a629-6a860b6db314-kube-api-access-cpjh6\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 00:59:59.969534 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.961138    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 00:59:59.969534 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.961283    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 00:59:59.969534 ip-10-0-130-169 kubenswrapper[5104]: I1030 00:59:59.964559    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.070609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.070573    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.070952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.070933    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.071130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.071117    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjh6\" (UniqueName: \"kubernetes.io/projected/d3dac020-aeba-427a-a629-6a860b6db314-kube-api-access-cpjh6\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.082936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.082893    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.109133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.109084    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"]
Oct 30 01:00:00.265779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.246253    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjh6\" (UniqueName: \"kubernetes.io/projected/d3dac020-aeba-427a-a629-6a860b6db314-kube-api-access-cpjh6\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.265779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.246512    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"]
Oct 30 01:00:00.481492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.462123    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b4144c2-pb877\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.481904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.481872    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 01:00:00.482086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.482060    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 01:00:00.486574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.486538    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:00.531156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.531058    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:00.597077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.594910    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"]
Oct 30 01:00:00.796537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.796061    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:00.918684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.918636    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:00.940484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.940419    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:00:00.940795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.940766    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 01:00:00.941285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.941245    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 01:00:00.941545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.941514    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 01:00:00.945029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.944997    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:00.961703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.961659    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:00.968595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.968242    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:00.970065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:00.968270    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:01.039587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.039539    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"]
Oct 30 01:00:01.086540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.057134    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:01.191912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.191875    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.389839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.383015    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rxv\" (UniqueName: \"kubernetes.io/projected/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-kube-api-access-m9rxv\") pod \"maas-api-key-cleanup-29888700-b2bd8\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.390049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.389896    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1634b743-68bc-4721-a409-dbafa459971b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:01.430493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.421818    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgfq\" (UniqueName: \"kubernetes.io/projected/1634b743-68bc-4721-a409-dbafa459971b-kube-api-access-jrgfq\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:01.466083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.461629    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 01:00:01.466083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.465589    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 01:00:01.466083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.465009    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:00:01.466083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.465796    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"]
Oct 30 01:00:01.469141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.468615    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-b2bd8\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.542596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.542564    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:00:01.542815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.542795    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n" event={"ID":"fef07710-514a-4672-b81c-682651eff83f","Type":"ContainerStarted","Data":"4dcdb9fdfa1176928390f5685a22535666668031b419d4c453157718e2a782cf"}
Oct 30 01:00:01.571476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.544583    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:01.571476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.547569    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"]
Oct 30 01:00:01.589552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.587931    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:01.646768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.646692    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"]
Oct 30 01:00:01.650607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.650563    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1634b743-68bc-4721-a409-dbafa459971b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:01.652128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.652100    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgfq\" (UniqueName: \"kubernetes.io/projected/1634b743-68bc-4721-a409-dbafa459971b-kube-api-access-jrgfq\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:01.652352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.652333    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-b2bd8\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.652512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.652491    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6936a14-59ad-4a85-89c4-fdcbc89444f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:01.653029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.652687    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1827e91-1568-4884-9142-3ef274a59c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:01.667071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.665726    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rxv\" (UniqueName: \"kubernetes.io/projected/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-kube-api-access-m9rxv\") pod \"maas-api-key-cleanup-29888700-b2bd8\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.675375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.672629    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mr6\" (UniqueName: \"kubernetes.io/projected/b6936a14-59ad-4a85-89c4-fdcbc89444f7-kube-api-access-z6mr6\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:01.677873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.677843    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6l8s\" (UniqueName: \"kubernetes.io/projected/f1827e91-1568-4884-9142-3ef274a59c89-kube-api-access-b6l8s\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:01.678844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.673244    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:00:01.680076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.680047    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 01:00:01.681721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.678093    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 01:00:01.699506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.699444    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 01:00:01.701769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.689710    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-b2bd8\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.701986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.651737    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1634b743-68bc-4721-a409-dbafa459971b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:01.739406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.739374    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rxv\" (UniqueName: \"kubernetes.io/projected/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-kube-api-access-m9rxv\") pod \"maas-api-key-cleanup-29888700-b2bd8\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.753144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.753111    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgfq\" (UniqueName: \"kubernetes.io/projected/1634b743-68bc-4721-a409-dbafa459971b-kube-api-access-jrgfq\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:01.819427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.819391    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6l8s\" (UniqueName: \"kubernetes.io/projected/f1827e91-1568-4884-9142-3ef274a59c89-kube-api-access-b6l8s\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:01.819897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.819868    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6936a14-59ad-4a85-89c4-fdcbc89444f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:01.820094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.820076    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1827e91-1568-4884-9142-3ef274a59c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:01.820236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.820220    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mr6\" (UniqueName: \"kubernetes.io/projected/b6936a14-59ad-4a85-89c4-fdcbc89444f7-kube-api-access-z6mr6\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:01.821900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.821877    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6936a14-59ad-4a85-89c4-fdcbc89444f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:01.822854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.822829    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1827e91-1568-4884-9142-3ef274a59c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:01.852184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.851880    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:01.860241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.859433    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6l8s\" (UniqueName: \"kubernetes.io/projected/f1827e91-1568-4884-9142-3ef274a59c89-kube-api-access-b6l8s\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:01.931473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.931422    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mr6\" (UniqueName: \"kubernetes.io/projected/b6936a14-59ad-4a85-89c4-fdcbc89444f7-kube-api-access-z6mr6\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:01.941666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.941637    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"]
Oct 30 01:00:01.954107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.954084    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"]
Oct 30 01:00:01.958980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.958901    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"]
Oct 30 01:00:01.966060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:01.966042    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"]
Oct 30 01:00:02.015541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.014355    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"]
Oct 30 01:00:02.017649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.017375    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:02.019250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.019233    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.135151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.134732    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwwx\" (UniqueName: \"kubernetes.io/projected/f79cb0d5-8682-4ecc-9152-72505eb43b9e-kube-api-access-zxwwx\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.135151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.134841    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.135151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.134971    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.156973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.156729    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:02.231750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.231393    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:02.241014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.237237    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.241014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.237382    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.241014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.237553    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwwx\" (UniqueName: \"kubernetes.io/projected/f79cb0d5-8682-4ecc-9152-72505eb43b9e-kube-api-access-zxwwx\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.241014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.239596    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.268088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.268052    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.273451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.273379    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwwx\" (UniqueName: \"kubernetes.io/projected/f79cb0d5-8682-4ecc-9152-72505eb43b9e-kube-api-access-zxwwx\") pod \"maas-api-revoke-sub-2db01cb1-qzfjc\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.321465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.321335    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"]
Oct 30 01:00:02.351505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.351217    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:02.400496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.400405    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 01:00:02.400711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.400495    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 01:00:02.455632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.455326    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"]
Oct 30 01:00:02.487830 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:02.485111    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17f8ea3_9bf4_4ee0_a8a3_74812fe632e9.slice/crio-04f8f3e208993630ddbf6843ee7b45cf986802389096ec2d4a2141c8d766ff5e WatchSource:0}: Error finding container 04f8f3e208993630ddbf6843ee7b45cf986802389096ec2d4a2141c8d766ff5e: Status 404 returned error can't find the container with id 04f8f3e208993630ddbf6843ee7b45cf986802389096ec2d4a2141c8d766ff5e
Oct 30 01:00:02.625927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.625604    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n" event={"ID":"fef07710-514a-4672-b81c-682651eff83f","Type":"ContainerStarted","Data":"9d22e45c1e1abddc2c73cde277b13a5c94d079e5c124b9144101066a5be6a13b"}
Oct 30 01:00:02.640990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.640880    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" event={"ID":"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9","Type":"ContainerStarted","Data":"04f8f3e208993630ddbf6843ee7b45cf986802389096ec2d4a2141c8d766ff5e"}
Oct 30 01:00:02.643655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.643502    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877" event={"ID":"d3dac020-aeba-427a-a629-6a860b6db314","Type":"ContainerStarted","Data":"22220122047fce5a3becff2e7fcbe87928a636acb25f507ccd6f87a413dc84e9"}
Oct 30 01:00:02.657031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.657003    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 01:00:02.657285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.657258    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 01:00:02.723529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.721557    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n" podStartSLOduration=6.721533716 podStartE2EDuration="6.721533716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.684270332 +0000 UTC m=+1044.231960828" watchObservedRunningTime="2026-10-30 01:00:02.721533716 +0000 UTC m=+1044.269224212"
Oct 30 01:00:02.723786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.723744    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"]
Oct 30 01:00:02.899565 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:02.899521    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6936a14_59ad_4a85_89c4_fdcbc89444f7.slice/crio-e3603df6cf55097fe09a94849bc4aec30894aa785780018d268583bc0e322416 WatchSource:0}: Error finding container e3603df6cf55097fe09a94849bc4aec30894aa785780018d268583bc0e322416: Status 404 returned error can't find the container with id e3603df6cf55097fe09a94849bc4aec30894aa785780018d268583bc0e322416
Oct 30 01:00:02.901241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.901213    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"]
Oct 30 01:00:02.925159 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:02.925124    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1827e91_1568_4884_9142_3ef274a59c89.slice/crio-0d8cb873b2ee4d782135087cb5c4383d9d163499b3b3045d1a778c405eab6c7b WatchSource:0}: Error finding container 0d8cb873b2ee4d782135087cb5c4383d9d163499b3b3045d1a778c405eab6c7b: Status 404 returned error can't find the container with id 0d8cb873b2ee4d782135087cb5c4383d9d163499b3b3045d1a778c405eab6c7b
Oct 30 01:00:02.928073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.928031    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"]
Oct 30 01:00:02.999745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:02.996162    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"]
Oct 30 01:00:03.403478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.399855    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 01:00:03.403478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.399931    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 01:00:03.541887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.538417    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 01:00:03.541887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.540012    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 01:00:03.541887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.541677    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 01:00:03.671492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.665854    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc" event={"ID":"f79cb0d5-8682-4ecc-9152-72505eb43b9e","Type":"ContainerStarted","Data":"9084ed1a0b51093158b477c3214ff5f8ad7dbed4843137cb78845d28386f6012"}
Oct 30 01:00:03.671492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.665920    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc" event={"ID":"f79cb0d5-8682-4ecc-9152-72505eb43b9e","Type":"ContainerStarted","Data":"a63370cd8ee6357c649ef9d66b69e5c64452c5123780df0ea24ef9fae01ced4e"}
Oct 30 01:00:03.671492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.668519    5104 generic.go:356] "Generic (PLEG): container finished" podID="fef07710-514a-4672-b81c-682651eff83f" containerID="9d22e45c1e1abddc2c73cde277b13a5c94d079e5c124b9144101066a5be6a13b" exitCode=0
Oct 30 01:00:03.671492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.668667    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n" event={"ID":"fef07710-514a-4672-b81c-682651eff83f","Type":"ContainerDied","Data":"9d22e45c1e1abddc2c73cde277b13a5c94d079e5c124b9144101066a5be6a13b"}
Oct 30 01:00:03.671492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.671045    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" event={"ID":"b6936a14-59ad-4a85-89c4-fdcbc89444f7","Type":"ContainerStarted","Data":"b73ad51a64bf7e1c1138c5d79539b28a1b844601919341048f3e045f509144f8"}
Oct 30 01:00:03.671492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.671098    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" event={"ID":"b6936a14-59ad-4a85-89c4-fdcbc89444f7","Type":"ContainerStarted","Data":"e3603df6cf55097fe09a94849bc4aec30894aa785780018d268583bc0e322416"}
Oct 30 01:00:03.677887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.676762    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" event={"ID":"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9","Type":"ContainerStarted","Data":"b64f01875a2f7c599f0d34426dc379bd1f93fcbde8d3a4a4e3fdbedcfa2ad151"}
Oct 30 01:00:03.684010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.683971    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" event={"ID":"f1827e91-1568-4884-9142-3ef274a59c89","Type":"ContainerStarted","Data":"5fcfce360670b9c0eddc76c331a3f9884f767b50752208c1b34b384522a3db7c"}
Oct 30 01:00:03.684243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.684222    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" event={"ID":"f1827e91-1568-4884-9142-3ef274a59c89","Type":"ContainerStarted","Data":"0d8cb873b2ee4d782135087cb5c4383d9d163499b3b3045d1a778c405eab6c7b"}
Oct 30 01:00:03.701519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.696979    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" event={"ID":"1634b743-68bc-4721-a409-dbafa459971b","Type":"ContainerStarted","Data":"a325323257ddcd805a4a9bfffa481096c8e7a51bd5ea5105541fb9539036c513"}
Oct 30 01:00:03.701781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.701749    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" event={"ID":"1634b743-68bc-4721-a409-dbafa459971b","Type":"ContainerStarted","Data":"6a94b555836705956e4d70a641543bf9c6de25016c5fe79fe4e31f8698adbebd"}
Oct 30 01:00:03.733934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.733844    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc" podStartSLOduration=3.733823756 podStartE2EDuration="3.733823756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.703124224 +0000 UTC m=+1045.250814721" watchObservedRunningTime="2026-10-30 01:00:03.733823756 +0000 UTC m=+1045.281514253"
Oct 30 01:00:03.787823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.787788    5104 generic.go:356] "Generic (PLEG): container finished" podID="d3dac020-aeba-427a-a629-6a860b6db314" containerID="65d1d0987626f31eb6ab6b6917935b63b9423c6cf5b0768343730fcb7ce1f336" exitCode=0
Oct 30 01:00:03.789528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.788905    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877" event={"ID":"d3dac020-aeba-427a-a629-6a860b6db314","Type":"ContainerDied","Data":"65d1d0987626f31eb6ab6b6917935b63b9423c6cf5b0768343730fcb7ce1f336"}
Oct 30 01:00:03.816848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.816788    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" podStartSLOduration=3.816769085 podStartE2EDuration="3.816769085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.780080245 +0000 UTC m=+1045.327770747" watchObservedRunningTime="2026-10-30 01:00:03.816769085 +0000 UTC m=+1045.364459582"
Oct 30 01:00:03.817647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.817608    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" podStartSLOduration=3.817597583 podStartE2EDuration="3.817597583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.815884223 +0000 UTC m=+1045.363574721" watchObservedRunningTime="2026-10-30 01:00:03.817597583 +0000 UTC m=+1045.365288085"
Oct 30 01:00:03.898671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.898608    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" podStartSLOduration=3.898587618 podStartE2EDuration="3.898587618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.89346771 +0000 UTC m=+1045.441158199" watchObservedRunningTime="2026-10-30 01:00:03.898587618 +0000 UTC m=+1045.446278116"
Oct 30 01:00:03.929569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.929502    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" podStartSLOduration=3.929482735 podStartE2EDuration="3.929482735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.928141923 +0000 UTC m=+1045.475832421" watchObservedRunningTime="2026-10-30 01:00:03.929482735 +0000 UTC m=+1045.477173233"
Oct 30 01:00:03.990504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.986802    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 01:00:03.990504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.988895    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 01:00:03.990504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:03.989911    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 01:00:04.404410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.404332    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 01:00:04.404913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.404875    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 01:00:04.405699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.405675    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 01:00:04.800891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.800858    5104 generic.go:356] "Generic (PLEG): container finished" podID="f79cb0d5-8682-4ecc-9152-72505eb43b9e" containerID="9084ed1a0b51093158b477c3214ff5f8ad7dbed4843137cb78845d28386f6012" exitCode=0
Oct 30 01:00:04.801081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.800942    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc" event={"ID":"f79cb0d5-8682-4ecc-9152-72505eb43b9e","Type":"ContainerDied","Data":"9084ed1a0b51093158b477c3214ff5f8ad7dbed4843137cb78845d28386f6012"}
Oct 30 01:00:04.803041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.803014    5104 generic.go:356] "Generic (PLEG): container finished" podID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerID="b73ad51a64bf7e1c1138c5d79539b28a1b844601919341048f3e045f509144f8" exitCode=60
Oct 30 01:00:04.803205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.803099    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" event={"ID":"b6936a14-59ad-4a85-89c4-fdcbc89444f7","Type":"ContainerDied","Data":"b73ad51a64bf7e1c1138c5d79539b28a1b844601919341048f3e045f509144f8"}
Oct 30 01:00:04.803431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.803415    5104 scope.go:122] "RemoveContainer" containerID="b73ad51a64bf7e1c1138c5d79539b28a1b844601919341048f3e045f509144f8"
Oct 30 01:00:04.806620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.806598    5104 generic.go:356] "Generic (PLEG): container finished" podID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerID="b64f01875a2f7c599f0d34426dc379bd1f93fcbde8d3a4a4e3fdbedcfa2ad151" exitCode=60
Oct 30 01:00:04.806838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.806635    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" event={"ID":"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9","Type":"ContainerDied","Data":"b64f01875a2f7c599f0d34426dc379bd1f93fcbde8d3a4a4e3fdbedcfa2ad151"}
Oct 30 01:00:04.806992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.806974    5104 scope.go:122] "RemoveContainer" containerID="b64f01875a2f7c599f0d34426dc379bd1f93fcbde8d3a4a4e3fdbedcfa2ad151"
Oct 30 01:00:04.812860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.812835    5104 generic.go:356] "Generic (PLEG): container finished" podID="f1827e91-1568-4884-9142-3ef274a59c89" containerID="5fcfce360670b9c0eddc76c331a3f9884f767b50752208c1b34b384522a3db7c" exitCode=60
Oct 30 01:00:04.813359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.813245    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" event={"ID":"f1827e91-1568-4884-9142-3ef274a59c89","Type":"ContainerDied","Data":"5fcfce360670b9c0eddc76c331a3f9884f767b50752208c1b34b384522a3db7c"}
Oct 30 01:00:04.813666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.813604    5104 scope.go:122] "RemoveContainer" containerID="5fcfce360670b9c0eddc76c331a3f9884f767b50752208c1b34b384522a3db7c"
Oct 30 01:00:04.823286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.823261    5104 generic.go:356] "Generic (PLEG): container finished" podID="1634b743-68bc-4721-a409-dbafa459971b" containerID="a325323257ddcd805a4a9bfffa481096c8e7a51bd5ea5105541fb9539036c513" exitCode=60
Oct 30 01:00:04.823393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.823361    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" event={"ID":"1634b743-68bc-4721-a409-dbafa459971b","Type":"ContainerDied","Data":"a325323257ddcd805a4a9bfffa481096c8e7a51bd5ea5105541fb9539036c513"}
Oct 30 01:00:04.823706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:04.823682    5104 scope.go:122] "RemoveContainer" containerID="a325323257ddcd805a4a9bfffa481096c8e7a51bd5ea5105541fb9539036c513"
Oct 30 01:00:05.236099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.236074    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 01:00:05.238780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.238755    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:05.271815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.271779    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef07710-514a-4672-b81c-682651eff83f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef07710-514a-4672-b81c-682651eff83f-maas-api-xks-ca\") pod \"fef07710-514a-4672-b81c-682651eff83f\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") "
Oct 30 01:00:05.272010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.271971    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-xks-ca\") pod \"d3dac020-aeba-427a-a629-6a860b6db314\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") "
Oct 30 01:00:05.272073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.272036    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3dac020-aeba-427a-a629-6a860b6db314-kube-api-access-cpjh6\" (UniqueName: \"kubernetes.io/projected/d3dac020-aeba-427a-a629-6a860b6db314-kube-api-access-cpjh6\") pod \"d3dac020-aeba-427a-a629-6a860b6db314\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") "
Oct 30 01:00:05.272136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.272123    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-openshift-ca\") pod \"d3dac020-aeba-427a-a629-6a860b6db314\" (UID: \"d3dac020-aeba-427a-a629-6a860b6db314\") "
Oct 30 01:00:05.272930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.272178    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef07710-514a-4672-b81c-682651eff83f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef07710-514a-4672-b81c-682651eff83f-maas-api-openshift-ca\") pod \"fef07710-514a-4672-b81c-682651eff83f\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") "
Oct 30 01:00:05.272930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.272233    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef07710-514a-4672-b81c-682651eff83f-kube-api-access-q5w46\" (UniqueName: \"kubernetes.io/projected/fef07710-514a-4672-b81c-682651eff83f-kube-api-access-q5w46\") pod \"fef07710-514a-4672-b81c-682651eff83f\" (UID: \"fef07710-514a-4672-b81c-682651eff83f\") "
Oct 30 01:00:05.273233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.273191    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-openshift-ca" pod "d3dac020-aeba-427a-a629-6a860b6db314" (UID: "d3dac020-aeba-427a-a629-6a860b6db314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:05.274997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.274915    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef07710-514a-4672-b81c-682651eff83f-maas-api-openshift-ca" pod "fef07710-514a-4672-b81c-682651eff83f" (UID: "fef07710-514a-4672-b81c-682651eff83f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:05.278384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.278320    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef07710-514a-4672-b81c-682651eff83f-maas-api-xks-ca" pod "fef07710-514a-4672-b81c-682651eff83f" (UID: "fef07710-514a-4672-b81c-682651eff83f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:05.278657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.278540    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3dac020-aeba-427a-a629-6a860b6db314-kube-api-access-cpjh6" pod "d3dac020-aeba-427a-a629-6a860b6db314" (UID: "d3dac020-aeba-427a-a629-6a860b6db314"). InnerVolumeSpecName "kube-api-access-cpjh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:05.279027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.278988    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-xks-ca" pod "d3dac020-aeba-427a-a629-6a860b6db314" (UID: "d3dac020-aeba-427a-a629-6a860b6db314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:05.280485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.280431    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef07710-514a-4672-b81c-682651eff83f-kube-api-access-q5w46" pod "fef07710-514a-4672-b81c-682651eff83f" (UID: "fef07710-514a-4672-b81c-682651eff83f"). InnerVolumeSpecName "kube-api-access-q5w46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:05.374856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.374819    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.374856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.374859    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef07710-514a-4672-b81c-682651eff83f-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.375086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.374873    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5w46\" (UniqueName: \"kubernetes.io/projected/fef07710-514a-4672-b81c-682651eff83f-kube-api-access-q5w46\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.375086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.374897    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef07710-514a-4672-b81c-682651eff83f-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.375086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.374912    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dac020-aeba-427a-a629-6a860b6db314-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.375086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.374924    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpjh6\" (UniqueName: \"kubernetes.io/projected/d3dac020-aeba-427a-a629-6a860b6db314-kube-api-access-cpjh6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.834442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.834403    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" event={"ID":"f1827e91-1568-4884-9142-3ef274a59c89","Type":"ContainerStarted","Data":"b9eaa03a18524473b8e64315a774dcab2dbfa4b7b2311a247aeed41bfc523e42"}
Oct 30 01:00:05.836603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.836575    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" event={"ID":"1634b743-68bc-4721-a409-dbafa459971b","Type":"ContainerStarted","Data":"9acaa50f89da379104890fccb0d96348c4d7b53ff22bfb91a2be4168d84a94bb"}
Oct 30 01:00:05.837994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.837968    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877" event={"ID":"d3dac020-aeba-427a-a629-6a860b6db314","Type":"ContainerDied","Data":"22220122047fce5a3becff2e7fcbe87928a636acb25f507ccd6f87a413dc84e9"}
Oct 30 01:00:05.837994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.837993    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"
Oct 30 01:00:05.838158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.838011    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22220122047fce5a3becff2e7fcbe87928a636acb25f507ccd6f87a413dc84e9"
Oct 30 01:00:05.839603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.839574    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n" event={"ID":"fef07710-514a-4672-b81c-682651eff83f","Type":"ContainerDied","Data":"4dcdb9fdfa1176928390f5685a22535666668031b419d4c453157718e2a782cf"}
Oct 30 01:00:05.839603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.839593    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"
Oct 30 01:00:05.839754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.839623    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcdb9fdfa1176928390f5685a22535666668031b419d4c453157718e2a782cf"
Oct 30 01:00:05.841772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.841606    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" event={"ID":"b6936a14-59ad-4a85-89c4-fdcbc89444f7","Type":"ContainerStarted","Data":"6b3e03e611fedc42a355445afb087af01b883b781d9982f0e0f49617853866d6"}
Oct 30 01:00:05.843809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.843779    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" event={"ID":"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9","Type":"ContainerStarted","Data":"885108cca4cc4a8870f31725996ebcc103297b2ac965400c09c82de5450a8113"}
Oct 30 01:00:05.980512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.979841    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"]
Oct 30 01:00:05.984676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.984650    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fef07710-514a-4672-b81c-682651eff83f" containerName="revoke-keys"
Oct 30 01:00:05.984878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.984858    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef07710-514a-4672-b81c-682651eff83f" containerName="revoke-keys"
Oct 30 01:00:05.985024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.985011    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d3dac020-aeba-427a-a629-6a860b6db314" containerName="revoke-keys"
Oct 30 01:00:05.985125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.985112    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3dac020-aeba-427a-a629-6a860b6db314" containerName="revoke-keys"
Oct 30 01:00:05.985596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.985579    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef07710-514a-4672-b81c-682651eff83f" containerName="revoke-keys"
Oct 30 01:00:05.985743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:05.985730    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3dac020-aeba-427a-a629-6a860b6db314" containerName="revoke-keys"
Oct 30 01:00:06.008901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.008843    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.014581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.014514    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"]
Oct 30 01:00:06.034136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.034108    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:06.090216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090145    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-xks-ca\") pod \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") "
Oct 30 01:00:06.090216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090185    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-openshift-ca\") pod \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") "
Oct 30 01:00:06.090420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090238    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79cb0d5-8682-4ecc-9152-72505eb43b9e-kube-api-access-zxwwx\" (UniqueName: \"kubernetes.io/projected/f79cb0d5-8682-4ecc-9152-72505eb43b9e-kube-api-access-zxwwx\") pod \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\" (UID: \"f79cb0d5-8682-4ecc-9152-72505eb43b9e\") "
Oct 30 01:00:06.090520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090501    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xdl\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-kube-api-access-d7xdl\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.090598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090553    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.090764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090741    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316e4b0-dc86-4cd1-9974-686040ba399f-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.090836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090821    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.090903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090869    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.090959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090900    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.090959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090907    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-openshift-ca" pod "f79cb0d5-8682-4ecc-9152-72505eb43b9e" (UID: "f79cb0d5-8682-4ecc-9152-72505eb43b9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:06.090959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.090929    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.091159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.091135    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.091515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.091236    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.091738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.091707    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:06.092415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.092384    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-xks-ca" pod "f79cb0d5-8682-4ecc-9152-72505eb43b9e" (UID: "f79cb0d5-8682-4ecc-9152-72505eb43b9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:06.092894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.092868    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79cb0d5-8682-4ecc-9152-72505eb43b9e-kube-api-access-zxwwx" pod "f79cb0d5-8682-4ecc-9152-72505eb43b9e" (UID: "f79cb0d5-8682-4ecc-9152-72505eb43b9e"). InnerVolumeSpecName "kube-api-access-zxwwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:06.193558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.193521    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.193759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.193678    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.193839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.193781    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xdl\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-kube-api-access-d7xdl\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.193839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.193816    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.193944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.193907    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316e4b0-dc86-4cd1-9974-686040ba399f-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.194031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.193974    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.194031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.194008    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.194293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.194019    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.194380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.194331    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.194380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.194367    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.194537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.194372    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.194537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.194526    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79cb0d5-8682-4ecc-9152-72505eb43b9e-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:06.194657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.194543    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwwx\" (UniqueName: \"kubernetes.io/projected/f79cb0d5-8682-4ecc-9152-72505eb43b9e-kube-api-access-zxwwx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:06.195285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.195258    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316e4b0-dc86-4cd1-9974-686040ba399f-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.195560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.195538    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.195720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.195683    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.196955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.196931    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.197499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.197479    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.212226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.212200    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.212386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.212364    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xdl\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-kube-api-access-d7xdl\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.330867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.330817    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:06.717840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.717802    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"]
Oct 30 01:00:06.718800 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:06.718773    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3316e4b0_dc86_4cd1_9974_686040ba399f.slice/crio-f90c6f64737f3911ef2be46952f3e4bd74b58dbd1a7ff2009516e08a1d4c204e WatchSource:0}: Error finding container f90c6f64737f3911ef2be46952f3e4bd74b58dbd1a7ff2009516e08a1d4c204e: Status 404 returned error can't find the container with id f90c6f64737f3911ef2be46952f3e4bd74b58dbd1a7ff2009516e08a1d4c204e
Oct 30 01:00:06.721790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.721755    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:06.721902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.721835    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:06.721902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.721881    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:06.855395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.855360    5104 generic.go:356] "Generic (PLEG): container finished" podID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerID="885108cca4cc4a8870f31725996ebcc103297b2ac965400c09c82de5450a8113" exitCode=60
Oct 30 01:00:06.855862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.855447    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" event={"ID":"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9","Type":"ContainerDied","Data":"885108cca4cc4a8870f31725996ebcc103297b2ac965400c09c82de5450a8113"}
Oct 30 01:00:06.855862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.855562    5104 scope.go:122] "RemoveContainer" containerID="b64f01875a2f7c599f0d34426dc379bd1f93fcbde8d3a4a4e3fdbedcfa2ad151"
Oct 30 01:00:06.855862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.855754    5104 scope.go:122] "RemoveContainer" containerID="885108cca4cc4a8870f31725996ebcc103297b2ac965400c09c82de5450a8113"
Oct 30 01:00:06.856176 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:06.856152    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-b2bd8_odh-ai-gateway-infra(c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9"
Oct 30 01:00:06.857596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.857570    5104 generic.go:356] "Generic (PLEG): container finished" podID="f1827e91-1568-4884-9142-3ef274a59c89" containerID="b9eaa03a18524473b8e64315a774dcab2dbfa4b7b2311a247aeed41bfc523e42" exitCode=60
Oct 30 01:00:06.857714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.857644    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" event={"ID":"f1827e91-1568-4884-9142-3ef274a59c89","Type":"ContainerDied","Data":"b9eaa03a18524473b8e64315a774dcab2dbfa4b7b2311a247aeed41bfc523e42"}
Oct 30 01:00:06.858093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.858073    5104 scope.go:122] "RemoveContainer" containerID="b9eaa03a18524473b8e64315a774dcab2dbfa4b7b2311a247aeed41bfc523e42"
Oct 30 01:00:06.858691 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:06.858430    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd_odh-ai-gateway-infra(f1827e91-1568-4884-9142-3ef274a59c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" podUID="f1827e91-1568-4884-9142-3ef274a59c89"
Oct 30 01:00:06.860495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.860201    5104 generic.go:356] "Generic (PLEG): container finished" podID="1634b743-68bc-4721-a409-dbafa459971b" containerID="9acaa50f89da379104890fccb0d96348c4d7b53ff22bfb91a2be4168d84a94bb" exitCode=60
Oct 30 01:00:06.860495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.860387    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" event={"ID":"1634b743-68bc-4721-a409-dbafa459971b","Type":"ContainerDied","Data":"9acaa50f89da379104890fccb0d96348c4d7b53ff22bfb91a2be4168d84a94bb"}
Oct 30 01:00:06.860910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.860576    5104 scope.go:122] "RemoveContainer" containerID="9acaa50f89da379104890fccb0d96348c4d7b53ff22bfb91a2be4168d84a94bb"
Oct 30 01:00:06.860972 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:06.860923    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4_odh-ai-gateway-infra(1634b743-68bc-4721-a409-dbafa459971b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" podUID="1634b743-68bc-4721-a409-dbafa459971b"
Oct 30 01:00:06.863521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.862655    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc" event={"ID":"f79cb0d5-8682-4ecc-9152-72505eb43b9e","Type":"ContainerDied","Data":"a63370cd8ee6357c649ef9d66b69e5c64452c5123780df0ea24ef9fae01ced4e"}
Oct 30 01:00:06.863521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.862697    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a63370cd8ee6357c649ef9d66b69e5c64452c5123780df0ea24ef9fae01ced4e"
Oct 30 01:00:06.863890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.863861    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"
Oct 30 01:00:06.864954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.864885    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" event={"ID":"3316e4b0-dc86-4cd1-9974-686040ba399f","Type":"ContainerStarted","Data":"04a942d802fe5764d9663d2058afea17e59e7026c7fea203ba1b500b7683632e"}
Oct 30 01:00:06.864954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.864926    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" event={"ID":"3316e4b0-dc86-4cd1-9974-686040ba399f","Type":"ContainerStarted","Data":"f90c6f64737f3911ef2be46952f3e4bd74b58dbd1a7ff2009516e08a1d4c204e"}
Oct 30 01:00:06.868685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.868661    5104 generic.go:356] "Generic (PLEG): container finished" podID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerID="6b3e03e611fedc42a355445afb087af01b883b781d9982f0e0f49617853866d6" exitCode=60
Oct 30 01:00:06.868780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.868696    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" event={"ID":"b6936a14-59ad-4a85-89c4-fdcbc89444f7","Type":"ContainerDied","Data":"6b3e03e611fedc42a355445afb087af01b883b781d9982f0e0f49617853866d6"}
Oct 30 01:00:06.869022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.869006    5104 scope.go:122] "RemoveContainer" containerID="6b3e03e611fedc42a355445afb087af01b883b781d9982f0e0f49617853866d6"
Oct 30 01:00:06.869313 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:06.869288    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv_odh-ai-gateway-infra(b6936a14-59ad-4a85-89c4-fdcbc89444f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7"
Oct 30 01:00:06.876293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.876270    5104 scope.go:122] "RemoveContainer" containerID="5fcfce360670b9c0eddc76c331a3f9884f767b50752208c1b34b384522a3db7c"
Oct 30 01:00:06.897358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.895854    5104 scope.go:122] "RemoveContainer" containerID="a325323257ddcd805a4a9bfffa481096c8e7a51bd5ea5105541fb9539036c513"
Oct 30 01:00:06.919591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:06.919558    5104 scope.go:122] "RemoveContainer" containerID="b73ad51a64bf7e1c1138c5d79539b28a1b844601919341048f3e045f509144f8"
Oct 30 01:00:07.018553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.018499    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podStartSLOduration=2.018482653 podStartE2EDuration="2.018482653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:07.017133672 +0000 UTC m=+1048.564824168" watchObservedRunningTime="2026-10-30 01:00:07.018482653 +0000 UTC m=+1048.566173149"
Oct 30 01:00:07.331728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.331634    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:07.333621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.333590    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:07.333760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.333661    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:07.913371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.913341    5104 scope.go:122] "RemoveContainer" containerID="6b3e03e611fedc42a355445afb087af01b883b781d9982f0e0f49617853866d6"
Oct 30 01:00:07.913884 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:07.913771    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv_odh-ai-gateway-infra(b6936a14-59ad-4a85-89c4-fdcbc89444f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7"
Oct 30 01:00:07.914987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.914963    5104 scope.go:122] "RemoveContainer" containerID="885108cca4cc4a8870f31725996ebcc103297b2ac965400c09c82de5450a8113"
Oct 30 01:00:07.915288 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:07.915265    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-b2bd8_odh-ai-gateway-infra(c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9"
Oct 30 01:00:07.916908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.916885    5104 scope.go:122] "RemoveContainer" containerID="b9eaa03a18524473b8e64315a774dcab2dbfa4b7b2311a247aeed41bfc523e42"
Oct 30 01:00:07.917284 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:07.917247    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd_odh-ai-gateway-infra(f1827e91-1568-4884-9142-3ef274a59c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" podUID="f1827e91-1568-4884-9142-3ef274a59c89"
Oct 30 01:00:07.918605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:07.918586    5104 scope.go:122] "RemoveContainer" containerID="9acaa50f89da379104890fccb0d96348c4d7b53ff22bfb91a2be4168d84a94bb"
Oct 30 01:00:07.918854 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:07.918834    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4_odh-ai-gateway-infra(1634b743-68bc-4721-a409-dbafa459971b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" podUID="1634b743-68bc-4721-a409-dbafa459971b"
Oct 30 01:00:08.332311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:08.332263    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:08.332536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:08.332329    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:09.136491 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:09.136426    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:00:09.331965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:09.331924    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:09.332129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:09.331984    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:09.820852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:09.820813    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:00:10.334484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:10.331945    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:10.334484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:10.332010    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:11.337476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:11.332912    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:11.337476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:11.332975    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:12.116491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.113599    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:12.131056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.130601    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:12.334487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.333973    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:12.334487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.334069    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:12.374543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.374298    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:12.401931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.400912    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:12.475809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.475756    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:12.531627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.531562    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:12.585626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.585578    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:12.682481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.681571    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:12.682481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.682006    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:12.705496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:12.702564    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:13.119546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.118553    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:13.119546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.119318    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:00:13.133655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.133609    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:13.134374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.134321    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:00:13.335490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.331875    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:13.335490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.331949    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:13.398541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.396599    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:13.398541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.397356    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:00:13.528484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.461123    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:13.528484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.515340    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:00:13.554102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.552897    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:00:13.554102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.553712    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:00:13.598332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.597814    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:13.611484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.610877    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:00:13.707806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.707766    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:00:13.708997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.708966    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:13.718479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.714646    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:00:13.718479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:13.714718    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:00:14.122282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.120114    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:14.147134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.147079    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:14.372537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.370742    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:14.372537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.370799    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:14.403104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.402743    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:14.480870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.480819    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:00:14.521601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.521549    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:14.557856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.553563    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:14.635541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.631568    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:14.749670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.749614    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:14.760096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:14.756560    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:15.125598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.125548    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.173636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.150811    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.361565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.361527    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:15.361875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.361846    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:15.395691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.395584    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:15.431486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.399595    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.432593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.425657    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:15.493927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.492260    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.532660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.532605    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:15.537500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.536013    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:00:15.565752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.565632    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.590686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.590627    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.768948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.768900    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:00:15.797592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.780647    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.797592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.797525    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.814875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.804557    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.887875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.845436    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:15.887875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.845524    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:15.905514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.900207    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:15.925633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:15.903535    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:16.491478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.207174    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:00:16.545554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.545502    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:00:16.566651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.566607    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:00:16.684704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.680154    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:16.684704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.683599    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:16.717096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.717048    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 01:00:16.717613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.717569    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:16.720296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.720258    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:16.818074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.800945    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:16.846484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.829301    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:16.875076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.875038    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:16.876615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:16.859267    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:00:17.353481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.353427    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:17.353781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.353747    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:17.454362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.454315    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:00:17.534763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.497070    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:17.575373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.548390    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:17.607754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.607647    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:17.705681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.705606    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:17.770478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.737863    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:17.938961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.741640    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:00:17.939434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:17.910646    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:18.109606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.109562    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:18.316128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.314800    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:00:18.371483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.358796    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 01:00:18.371483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.358857    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 01:00:18.504476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.454863    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"]
Oct 30 01:00:18.504476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.456102    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f79cb0d5-8682-4ecc-9152-72505eb43b9e" containerName="revoke-keys"
Oct 30 01:00:18.504476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.456127    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79cb0d5-8682-4ecc-9152-72505eb43b9e" containerName="revoke-keys"
Oct 30 01:00:18.504476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.456314    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79cb0d5-8682-4ecc-9152-72505eb43b9e" containerName="revoke-keys"
Oct 30 01:00:18.535481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.515543    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:18.570327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.558870    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Oct 30 01:00:18.602477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.599182    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:18.602477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.601246    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:00:18.649020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.648975    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:18.669449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.669417    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.696637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.674204    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:00:18.696637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.677620    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:00:18.696637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.677656    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:00:18.696637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.682467    5104 scope.go:122] "RemoveContainer" containerID="9acaa50f89da379104890fccb0d96348c4d7b53ff22bfb91a2be4168d84a94bb"
Oct 30 01:00:18.697849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.697826    5104 scope.go:122] "RemoveContainer" containerID="6b3e03e611fedc42a355445afb087af01b883b781d9982f0e0f49617853866d6"
Oct 30 01:00:18.746754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.717879    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.758068    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.758136    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhdp\" (UniqueName: \"kubernetes.io/projected/c423303f-001a-43c2-96bf-2d5377abe4ef-kube-api-access-nwhdp\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.758204    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.764606    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"]
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.764695    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.764742    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.764774    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.764802    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:00:18.765575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.764901    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:00:18.860665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.860628    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhdp\" (UniqueName: \"kubernetes.io/projected/c423303f-001a-43c2-96bf-2d5377abe4ef-kube-api-access-nwhdp\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.860871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.860786    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.861894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.861861    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.866940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.866910    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.869329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.869303    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:18.885534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:18.884341    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhdp\" (UniqueName: \"kubernetes.io/projected/c423303f-001a-43c2-96bf-2d5377abe4ef-kube-api-access-nwhdp\") pod \"maas-api-revoke-sub-bba58e55-fxrdq\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:19.083546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.083437    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:19.507436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.507398    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"]
Oct 30 01:00:19.513829 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:19.513801    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc423303f_001a_43c2_96bf_2d5377abe4ef.slice/crio-7e635b9e01cc82c64560bf533a02285939bbe08f97ba179d260b25f616ed1a44 WatchSource:0}: Error finding container 7e635b9e01cc82c64560bf533a02285939bbe08f97ba179d260b25f616ed1a44: Status 404 returned error can't find the container with id 7e635b9e01cc82c64560bf533a02285939bbe08f97ba179d260b25f616ed1a44
Oct 30 01:00:19.569595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.569556    5104 scope.go:122] "RemoveContainer" containerID="885108cca4cc4a8870f31725996ebcc103297b2ac965400c09c82de5450a8113"
Oct 30 01:00:19.599865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.597045    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:19.599865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.599111    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:19.600982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.600587    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:00:19.968574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.968534    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq" event={"ID":"c423303f-001a-43c2-96bf-2d5377abe4ef","Type":"ContainerStarted","Data":"7e635b9e01cc82c64560bf533a02285939bbe08f97ba179d260b25f616ed1a44"}
Oct 30 01:00:19.971600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.971552    5104 generic.go:356] "Generic (PLEG): container finished" podID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerID="180152d227e9582dec2d44bdcf04b459a2a48007adc9f83a370e9c06e0ca0797" exitCode=60
Oct 30 01:00:19.971949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.971921    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" event={"ID":"b6936a14-59ad-4a85-89c4-fdcbc89444f7","Type":"ContainerDied","Data":"180152d227e9582dec2d44bdcf04b459a2a48007adc9f83a370e9c06e0ca0797"}
Oct 30 01:00:19.972201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.972148    5104 scope.go:122] "RemoveContainer" containerID="6b3e03e611fedc42a355445afb087af01b883b781d9982f0e0f49617853866d6"
Oct 30 01:00:19.976479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:19.972866    5104 scope.go:122] "RemoveContainer" containerID="180152d227e9582dec2d44bdcf04b459a2a48007adc9f83a370e9c06e0ca0797"
Oct 30 01:00:19.986479 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:19.977033    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv_odh-ai-gateway-infra(b6936a14-59ad-4a85-89c4-fdcbc89444f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7"
Oct 30 01:00:20.015391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:20.015360    5104 generic.go:356] "Generic (PLEG): container finished" podID="1634b743-68bc-4721-a409-dbafa459971b" containerID="7b40caab2b120259c37a24c5bebc3a4bfa6bd5d1592c07c42aa66700ed43745b" exitCode=60
Oct 30 01:00:20.015784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:20.015554    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" event={"ID":"1634b743-68bc-4721-a409-dbafa459971b","Type":"ContainerDied","Data":"7b40caab2b120259c37a24c5bebc3a4bfa6bd5d1592c07c42aa66700ed43745b"}
Oct 30 01:00:20.018973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:20.018949    5104 scope.go:122] "RemoveContainer" containerID="7b40caab2b120259c37a24c5bebc3a4bfa6bd5d1592c07c42aa66700ed43745b"
Oct 30 01:00:20.019867 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:20.019834    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4_odh-ai-gateway-infra(1634b743-68bc-4721-a409-dbafa459971b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" podUID="1634b743-68bc-4721-a409-dbafa459971b"
Oct 30 01:00:20.079523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:20.079367    5104 scope.go:122] "RemoveContainer" containerID="9acaa50f89da379104890fccb0d96348c4d7b53ff22bfb91a2be4168d84a94bb"
Oct 30 01:00:20.570365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:20.570338    5104 scope.go:122] "RemoveContainer" containerID="b9eaa03a18524473b8e64315a774dcab2dbfa4b7b2311a247aeed41bfc523e42"
Oct 30 01:00:21.012623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.012587    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"]
Oct 30 01:00:21.018557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.018530    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.021835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.021808    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.021985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.021852    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.021985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.021880    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.021985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.021950    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.022164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.021993    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.022164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.022020    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hzk\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-kube-api-access-h6hzk\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.022164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.022148    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.022329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.022189    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.022329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.022221    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.041813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.041768    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"]
Oct 30 01:00:21.068866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.068827    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"]
Oct 30 01:00:21.071362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.071332    5104 generic.go:356] "Generic (PLEG): container finished" podID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerID="c4ddb7050d8f4f934b68209bb2dee1fd0fb391dc4b8b951311af12be913c2351" exitCode=60
Oct 30 01:00:21.072056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.071444    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" event={"ID":"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9","Type":"ContainerDied","Data":"c4ddb7050d8f4f934b68209bb2dee1fd0fb391dc4b8b951311af12be913c2351"}
Oct 30 01:00:21.072056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.071547    5104 scope.go:122] "RemoveContainer" containerID="885108cca4cc4a8870f31725996ebcc103297b2ac965400c09c82de5450a8113"
Oct 30 01:00:21.072278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.072097    5104 scope.go:122] "RemoveContainer" containerID="c4ddb7050d8f4f934b68209bb2dee1fd0fb391dc4b8b951311af12be913c2351"
Oct 30 01:00:21.081675 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:21.072516    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-b2bd8_odh-ai-gateway-infra(c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9"
Oct 30 01:00:21.081675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.078498    5104 generic.go:356] "Generic (PLEG): container finished" podID="c423303f-001a-43c2-96bf-2d5377abe4ef" containerID="b73222d179b5c4346b4d211db98fea5849ead8bb0e3b84b2ce55bac62f5091dc" exitCode=0
Oct 30 01:00:21.081675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.079612    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq" event={"ID":"c423303f-001a-43c2-96bf-2d5377abe4ef","Type":"ContainerDied","Data":"b73222d179b5c4346b4d211db98fea5849ead8bb0e3b84b2ce55bac62f5091dc"}
Oct 30 01:00:21.116550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.106195    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"]
Oct 30 01:00:21.124292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124249    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124306    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124335    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124497    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124541    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124564    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124661    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124710    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.124784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.124733    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hzk\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-kube-api-access-h6hzk\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.125754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.125730    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.126358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.126335    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.130285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.130259    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.133355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.133328    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.133880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.133813    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.134293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.134270    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.165622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.165553    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hzk\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-kube-api-access-h6hzk\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.168188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.168135    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.172654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.172618    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.336338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.336260    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"]
Oct 30 01:00:21.340385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.340352    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:21.343124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.343092    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:21.344738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.344713    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.372519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.371584    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"]
Oct 30 01:00:21.422197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.422163    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:21.430170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.430136    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1634b743-68bc-4721-a409-dbafa459971b-kube-api-access-jrgfq\" (UniqueName: \"kubernetes.io/projected/1634b743-68bc-4721-a409-dbafa459971b-kube-api-access-jrgfq\") pod \"1634b743-68bc-4721-a409-dbafa459971b\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") "
Oct 30 01:00:21.430450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.430426    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1634b743-68bc-4721-a409-dbafa459971b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1634b743-68bc-4721-a409-dbafa459971b-maas-api-service-ca\") pod \"1634b743-68bc-4721-a409-dbafa459971b\" (UID: \"1634b743-68bc-4721-a409-dbafa459971b\") "
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.437550    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6936a14-59ad-4a85-89c4-fdcbc89444f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6936a14-59ad-4a85-89c4-fdcbc89444f7-maas-api-service-ca\") pod \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") "
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.435533    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1634b743-68bc-4721-a409-dbafa459971b-kube-api-access-jrgfq" pod "1634b743-68bc-4721-a409-dbafa459971b" (UID: "1634b743-68bc-4721-a409-dbafa459971b"). InnerVolumeSpecName "kube-api-access-jrgfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.437626    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6936a14-59ad-4a85-89c4-fdcbc89444f7-kube-api-access-z6mr6\" (UniqueName: \"kubernetes.io/projected/b6936a14-59ad-4a85-89c4-fdcbc89444f7-kube-api-access-z6mr6\") pod \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\" (UID: \"b6936a14-59ad-4a85-89c4-fdcbc89444f7\") "
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.438195    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6936a14-59ad-4a85-89c4-fdcbc89444f7-maas-api-service-ca" pod "b6936a14-59ad-4a85-89c4-fdcbc89444f7" (UID: "b6936a14-59ad-4a85-89c4-fdcbc89444f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.438301    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.438448    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972nw\" (UniqueName: \"kubernetes.io/projected/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-kube-api-access-972nw\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.438535    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.438645    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrgfq\" (UniqueName: \"kubernetes.io/projected/1634b743-68bc-4721-a409-dbafa459971b-kube-api-access-jrgfq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.438661    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6936a14-59ad-4a85-89c4-fdcbc89444f7-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.439014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.438882    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1634b743-68bc-4721-a409-dbafa459971b-maas-api-service-ca" pod "1634b743-68bc-4721-a409-dbafa459971b" (UID: "1634b743-68bc-4721-a409-dbafa459971b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:21.441828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.440798    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6936a14-59ad-4a85-89c4-fdcbc89444f7-kube-api-access-z6mr6" pod "b6936a14-59ad-4a85-89c4-fdcbc89444f7" (UID: "b6936a14-59ad-4a85-89c4-fdcbc89444f7"). InnerVolumeSpecName "kube-api-access-z6mr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:21.540318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.540266    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.540764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.540739    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972nw\" (UniqueName: \"kubernetes.io/projected/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-kube-api-access-972nw\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.540971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.540954    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.541161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.541128    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.541377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.541360    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1634b743-68bc-4721-a409-dbafa459971b-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.541533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.541518    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mr6\" (UniqueName: \"kubernetes.io/projected/b6936a14-59ad-4a85-89c4-fdcbc89444f7-kube-api-access-z6mr6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.543250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.543229    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.559294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.559256    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972nw\" (UniqueName: \"kubernetes.io/projected/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-kube-api-access-972nw\") pod \"maas-api-revoke-sub-9b7bb922-77d5q\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.683663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.683630    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:21.707416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.707391    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"]
Oct 30 01:00:21.711028 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:21.710993    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a4cd1e_7da7_44a9_aae6_8d1e7e9064aa.slice/crio-731b2dba109ae9d123c7fc174e01aad2d321461da071b70b2793454d2069f6f3 WatchSource:0}: Error finding container 731b2dba109ae9d123c7fc174e01aad2d321461da071b70b2793454d2069f6f3: Status 404 returned error can't find the container with id 731b2dba109ae9d123c7fc174e01aad2d321461da071b70b2793454d2069f6f3
Oct 30 01:00:21.713858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.713822    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:21.713961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.713899    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:21.713961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:21.713945    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:22.014640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.013842    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"]
Oct 30 01:00:22.092989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.092963    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"
Oct 30 01:00:22.093182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.092958    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4" event={"ID":"1634b743-68bc-4721-a409-dbafa459971b","Type":"ContainerDied","Data":"6a94b555836705956e4d70a641543bf9c6de25016c5fe79fe4e31f8698adbebd"}
Oct 30 01:00:22.093182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.093068    5104 scope.go:122] "RemoveContainer" containerID="7b40caab2b120259c37a24c5bebc3a4bfa6bd5d1592c07c42aa66700ed43745b"
Oct 30 01:00:22.095410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.095380    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q" event={"ID":"84e0f99b-241d-4128-a42a-5a5dc24ea4fa","Type":"ContainerStarted","Data":"68661c416015b5106c77e9055644c4ef4ecebde526ab944560c3f45917417d82"}
Oct 30 01:00:22.097557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.097526    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"
Oct 30 01:00:22.097696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.097533    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv" event={"ID":"b6936a14-59ad-4a85-89c4-fdcbc89444f7","Type":"ContainerDied","Data":"e3603df6cf55097fe09a94849bc4aec30894aa785780018d268583bc0e322416"}
Oct 30 01:00:22.099952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.099931    5104 generic.go:356] "Generic (PLEG): container finished" podID="f1827e91-1568-4884-9142-3ef274a59c89" containerID="6689965b08a871757be72704f90b543a9ecaddc240a62e539031c951b704d236" exitCode=60
Oct 30 01:00:22.100054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.099998    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" event={"ID":"f1827e91-1568-4884-9142-3ef274a59c89","Type":"ContainerDied","Data":"6689965b08a871757be72704f90b543a9ecaddc240a62e539031c951b704d236"}
Oct 30 01:00:22.100383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.100368    5104 scope.go:122] "RemoveContainer" containerID="6689965b08a871757be72704f90b543a9ecaddc240a62e539031c951b704d236"
Oct 30 01:00:22.100758 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:22.100733    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd_odh-ai-gateway-infra(f1827e91-1568-4884-9142-3ef274a59c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" podUID="f1827e91-1568-4884-9142-3ef274a59c89"
Oct 30 01:00:22.102211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.102189    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" event={"ID":"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa","Type":"ContainerStarted","Data":"7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f"}
Oct 30 01:00:22.102315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.102230    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" event={"ID":"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa","Type":"ContainerStarted","Data":"731b2dba109ae9d123c7fc174e01aad2d321461da071b70b2793454d2069f6f3"}
Oct 30 01:00:22.108372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.108352    5104 scope.go:122] "RemoveContainer" containerID="180152d227e9582dec2d44bdcf04b459a2a48007adc9f83a370e9c06e0ca0797"
Oct 30 01:00:22.124145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.123760    5104 scope.go:122] "RemoveContainer" containerID="b9eaa03a18524473b8e64315a774dcab2dbfa4b7b2311a247aeed41bfc523e42"
Oct 30 01:00:22.165112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.165060    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podStartSLOduration=2.165044694 podStartE2EDuration="2.165044694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:22.16102059 +0000 UTC m=+1063.708711087" watchObservedRunningTime="2026-10-30 01:00:22.165044694 +0000 UTC m=+1063.712735258"
Oct 30 01:00:22.219861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.218069    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"]
Oct 30 01:00:22.259087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.259051    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"]
Oct 30 01:00:22.268915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.268834    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888700-2bcs4"]
Oct 30 01:00:22.301174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.301073    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"]
Oct 30 01:00:22.302213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.302180    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888700-mbwxv"]
Oct 30 01:00:22.333217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.333190    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:22.346515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.344143    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:22.346515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.345257    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:00:22.346515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.345333    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:00:22.360792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.360745    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-openshift-ca\") pod \"c423303f-001a-43c2-96bf-2d5377abe4ef\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") "
Oct 30 01:00:22.361187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.361164    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-xks-ca\") pod \"c423303f-001a-43c2-96bf-2d5377abe4ef\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") "
Oct 30 01:00:22.361368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.361354    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c423303f-001a-43c2-96bf-2d5377abe4ef-kube-api-access-nwhdp\" (UniqueName: \"kubernetes.io/projected/c423303f-001a-43c2-96bf-2d5377abe4ef-kube-api-access-nwhdp\") pod \"c423303f-001a-43c2-96bf-2d5377abe4ef\" (UID: \"c423303f-001a-43c2-96bf-2d5377abe4ef\") "
Oct 30 01:00:22.362198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.362167    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-openshift-ca" pod "c423303f-001a-43c2-96bf-2d5377abe4ef" (UID: "c423303f-001a-43c2-96bf-2d5377abe4ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.366972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.366935    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-xks-ca" pod "c423303f-001a-43c2-96bf-2d5377abe4ef" (UID: "c423303f-001a-43c2-96bf-2d5377abe4ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:22.367998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.367976    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c423303f-001a-43c2-96bf-2d5377abe4ef-kube-api-access-nwhdp" pod "c423303f-001a-43c2-96bf-2d5377abe4ef" (UID: "c423303f-001a-43c2-96bf-2d5377abe4ef"). InnerVolumeSpecName "kube-api-access-nwhdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:22.477868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.477831    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.479570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.479545    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhdp\" (UniqueName: \"kubernetes.io/projected/c423303f-001a-43c2-96bf-2d5377abe4ef-kube-api-access-nwhdp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.479767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.479752    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c423303f-001a-43c2-96bf-2d5377abe4ef-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.492713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.492674    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:22.614275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.614190    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1634b743-68bc-4721-a409-dbafa459971b" path="/var/lib/kubelet/pods/1634b743-68bc-4721-a409-dbafa459971b/volumes"
Oct 30 01:00:22.614846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.614821    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" path="/var/lib/kubelet/pods/b6936a14-59ad-4a85-89c4-fdcbc89444f7/volumes"
Oct 30 01:00:22.699588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.699528    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-kube-api-access-m9rxv\" (UniqueName: \"kubernetes.io/projected/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-kube-api-access-m9rxv\") pod \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") "
Oct 30 01:00:22.700066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.699811    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-maas-api-service-ca\") pod \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\" (UID: \"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9\") "
Oct 30 01:00:22.700986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.700942    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-maas-api-service-ca" pod "c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" (UID: "c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.702911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.702884    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-kube-api-access-m9rxv" pod "c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" (UID: "c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9"). InnerVolumeSpecName "kube-api-access-m9rxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:22.802335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.802301    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.802335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:22.802332    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rxv\" (UniqueName: \"kubernetes.io/projected/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9-kube-api-access-m9rxv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.118994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.118957    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"
Oct 30 01:00:23.118994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.118985    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq" event={"ID":"c423303f-001a-43c2-96bf-2d5377abe4ef","Type":"ContainerDied","Data":"7e635b9e01cc82c64560bf533a02285939bbe08f97ba179d260b25f616ed1a44"}
Oct 30 01:00:23.119195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.119041    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e635b9e01cc82c64560bf533a02285939bbe08f97ba179d260b25f616ed1a44"
Oct 30 01:00:23.120588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.120569    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"
Oct 30 01:00:23.120800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.120577    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8" event={"ID":"c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9","Type":"ContainerDied","Data":"04f8f3e208993630ddbf6843ee7b45cf986802389096ec2d4a2141c8d766ff5e"}
Oct 30 01:00:23.120884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.120843    5104 scope.go:122] "RemoveContainer" containerID="c4ddb7050d8f4f934b68209bb2dee1fd0fb391dc4b8b951311af12be913c2351"
Oct 30 01:00:23.126266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.126238    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q" event={"ID":"84e0f99b-241d-4128-a42a-5a5dc24ea4fa","Type":"ContainerStarted","Data":"bc3269130ebfaa514af493c84a3cf12048cb97090c27f67a89f3d70a744da1a9"}
Oct 30 01:00:23.209202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.209165    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"]
Oct 30 01:00:23.216531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.216499    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"]
Oct 30 01:00:23.228581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.228549    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-b2bd8"]
Oct 30 01:00:23.344530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.344493    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:00:23.344711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.344558    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:00:23.462022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.461988    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:23.515082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.515045    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1827e91-1568-4884-9142-3ef274a59c89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1827e91-1568-4884-9142-3ef274a59c89-maas-api-service-ca\") pod \"f1827e91-1568-4884-9142-3ef274a59c89\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") "
Oct 30 01:00:23.515556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.515508    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1827e91-1568-4884-9142-3ef274a59c89-kube-api-access-b6l8s\" (UniqueName: \"kubernetes.io/projected/f1827e91-1568-4884-9142-3ef274a59c89-kube-api-access-b6l8s\") pod \"f1827e91-1568-4884-9142-3ef274a59c89\" (UID: \"f1827e91-1568-4884-9142-3ef274a59c89\") "
Oct 30 01:00:23.515920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.515889    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1827e91-1568-4884-9142-3ef274a59c89-maas-api-service-ca" pod "f1827e91-1568-4884-9142-3ef274a59c89" (UID: "f1827e91-1568-4884-9142-3ef274a59c89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:23.516397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.516371    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1827e91-1568-4884-9142-3ef274a59c89-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.517979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.517957    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1827e91-1568-4884-9142-3ef274a59c89-kube-api-access-b6l8s" pod "f1827e91-1568-4884-9142-3ef274a59c89" (UID: "f1827e91-1568-4884-9142-3ef274a59c89"). InnerVolumeSpecName "kube-api-access-b6l8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:23.647110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:23.646613    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6l8s\" (UniqueName: \"kubernetes.io/projected/f1827e91-1568-4884-9142-3ef274a59c89-kube-api-access-b6l8s\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.140560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.140522    5104 generic.go:356] "Generic (PLEG): container finished" podID="84e0f99b-241d-4128-a42a-5a5dc24ea4fa" containerID="bc3269130ebfaa514af493c84a3cf12048cb97090c27f67a89f3d70a744da1a9" exitCode=0
Oct 30 01:00:24.141016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.140601    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q" event={"ID":"84e0f99b-241d-4128-a42a-5a5dc24ea4fa","Type":"ContainerDied","Data":"bc3269130ebfaa514af493c84a3cf12048cb97090c27f67a89f3d70a744da1a9"}
Oct 30 01:00:24.142257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.142229    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd" event={"ID":"f1827e91-1568-4884-9142-3ef274a59c89","Type":"ContainerDied","Data":"0d8cb873b2ee4d782135087cb5c4383d9d163499b3b3045d1a778c405eab6c7b"}
Oct 30 01:00:24.142257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.142246    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"
Oct 30 01:00:24.142423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.142291    5104 scope.go:122] "RemoveContainer" containerID="6689965b08a871757be72704f90b543a9ecaddc240a62e539031c951b704d236"
Oct 30 01:00:24.206632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.206549    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"]
Oct 30 01:00:24.211736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.211709    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888700-nslpd"]
Oct 30 01:00:24.344514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.344486    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:00:24.344660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.344547    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:00:24.366806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.366777    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:24.478479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.477893    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-kube-api-access-972nw\" (UniqueName: \"kubernetes.io/projected/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-kube-api-access-972nw\") pod \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") "
Oct 30 01:00:24.478479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.478074    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-openshift-ca\") pod \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") "
Oct 30 01:00:24.478479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.478114    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-xks-ca\") pod \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\" (UID: \"84e0f99b-241d-4128-a42a-5a5dc24ea4fa\") "
Oct 30 01:00:24.517346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.503669    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-openshift-ca" pod "84e0f99b-241d-4128-a42a-5a5dc24ea4fa" (UID: "84e0f99b-241d-4128-a42a-5a5dc24ea4fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:24.517346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.513196    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-xks-ca" pod "84e0f99b-241d-4128-a42a-5a5dc24ea4fa" (UID: "84e0f99b-241d-4128-a42a-5a5dc24ea4fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:24.518649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.518073    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-kube-api-access-972nw" pod "84e0f99b-241d-4128-a42a-5a5dc24ea4fa" (UID: "84e0f99b-241d-4128-a42a-5a5dc24ea4fa"). InnerVolumeSpecName "kube-api-access-972nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:24.612480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.611697    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972nw\" (UniqueName: \"kubernetes.io/projected/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-kube-api-access-972nw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.612480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.611938    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.612743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.612632    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e0f99b-241d-4128-a42a-5a5dc24ea4fa-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:24.625166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.625127    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" path="/var/lib/kubelet/pods/c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9/volumes"
Oct 30 01:00:24.625711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:24.625689    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1827e91-1568-4884-9142-3ef274a59c89" path="/var/lib/kubelet/pods/f1827e91-1568-4884-9142-3ef274a59c89/volumes"
Oct 30 01:00:25.152011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.151986    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"
Oct 30 01:00:25.152408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.152025    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q" event={"ID":"84e0f99b-241d-4128-a42a-5a5dc24ea4fa","Type":"ContainerDied","Data":"68661c416015b5106c77e9055644c4ef4ecebde526ab944560c3f45917417d82"}
Oct 30 01:00:25.152408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.152087    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68661c416015b5106c77e9055644c4ef4ecebde526ab944560c3f45917417d82"
Oct 30 01:00:25.344027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.343993    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:00:25.344183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.344057    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:00:25.405808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.405706    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"]
Oct 30 01:00:25.406753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406733    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.406846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406754    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.406846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406775    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:25.406846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406784    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:25.406846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406816    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:25.406846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406823    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:25.406846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406833    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.406846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406840    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406858    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406865    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406873    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406880    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406890    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406897    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406909    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406915    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406934    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406941    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406955    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c423303f-001a-43c2-96bf-2d5377abe4ef" containerName="revoke-keys"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406962    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c423303f-001a-43c2-96bf-2d5377abe4ef" containerName="revoke-keys"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406985    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.406991    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407002    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407008    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407047    5104 cpu_manager.go:414] "containerMap: removing container" podUID="84e0f99b-241d-4128-a42a-5a5dc24ea4fa" containerName="revoke-keys"
Oct 30 01:00:25.407051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407054    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e0f99b-241d-4128-a42a-5a5dc24ea4fa" containerName="revoke-keys"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407193    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407207    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407221    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407231    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407238    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1634b743-68bc-4721-a409-dbafa459971b" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407248    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407256    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407267    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6936a14-59ad-4a85-89c4-fdcbc89444f7" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407278    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c423303f-001a-43c2-96bf-2d5377abe4ef" containerName="revoke-keys"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407291    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407301    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1827e91-1568-4884-9142-3ef274a59c89" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407310    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:25.407550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.407319    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e0f99b-241d-4128-a42a-5a5dc24ea4fa" containerName="revoke-keys"
Oct 30 01:00:25.411861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.411835    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.426070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.425384    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.426070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.425537    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.426070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.425666    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:25.426070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.425699    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7sf\" (UniqueName: \"kubernetes.io/projected/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-kube-api-access-mf7sf\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.426070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.425753    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:25.426824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.426799    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"]
Oct 30 01:00:25.456493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.453742    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:00:25.456493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.453820    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:25.573484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.572848    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7sf\" (UniqueName: \"kubernetes.io/projected/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-kube-api-access-mf7sf\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.573484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.572982    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.573484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.573085    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.600478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.573975    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.641136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.639064    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.641136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.641027    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7sf\" (UniqueName: \"kubernetes.io/projected/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-kube-api-access-mf7sf\") pod \"maas-api-revoke-sub-756ff380-xrdhb\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:25.728041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:25.728007    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:26.070508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:26.070475    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"]
Oct 30 01:00:26.071151 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:26.071122    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6187ebe3_e4f8_401f_93a8_e9d60226b8f7.slice/crio-62bfa378ba5208ec5f496923fd0258b229265c80f5d32f87667e9e8058a32f3b WatchSource:0}: Error finding container 62bfa378ba5208ec5f496923fd0258b229265c80f5d32f87667e9e8058a32f3b: Status 404 returned error can't find the container with id 62bfa378ba5208ec5f496923fd0258b229265c80f5d32f87667e9e8058a32f3b
Oct 30 01:00:26.161473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:26.161419    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb" event={"ID":"6187ebe3-e4f8-401f-93a8-e9d60226b8f7","Type":"ContainerStarted","Data":"62bfa378ba5208ec5f496923fd0258b229265c80f5d32f87667e9e8058a32f3b"}
Oct 30 01:00:26.344475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:26.344379    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:00:26.344647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:26.344490    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:00:27.172522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:27.172423    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb" event={"ID":"6187ebe3-e4f8-401f-93a8-e9d60226b8f7","Type":"ContainerStarted","Data":"5e7bed58f7b6aa65b71fa259e46b428ad35ebb65c15fe72568a50be45e7b3712"}
Oct 30 01:00:27.231725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:27.231676    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb" podStartSLOduration=2.2316631 podStartE2EDuration="2.2316631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:27.231575353 +0000 UTC m=+1068.779265849" watchObservedRunningTime="2026-10-30 01:00:27.2316631 +0000 UTC m=+1068.779353593"
Oct 30 01:00:27.353057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:27.345857    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:00:27.353057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:27.345927    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:00:28.183289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:28.183246    5104 generic.go:356] "Generic (PLEG): container finished" podID="6187ebe3-e4f8-401f-93a8-e9d60226b8f7" containerID="5e7bed58f7b6aa65b71fa259e46b428ad35ebb65c15fe72568a50be45e7b3712" exitCode=0
Oct 30 01:00:28.183745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:28.183336    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb" event={"ID":"6187ebe3-e4f8-401f-93a8-e9d60226b8f7","Type":"ContainerDied","Data":"5e7bed58f7b6aa65b71fa259e46b428ad35ebb65c15fe72568a50be45e7b3712"}
Oct 30 01:00:28.343550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:28.343514    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:00:28.343726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:28.343568    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:00:29.407427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.407396    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:29.485049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.484942    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-xks-ca\") pod \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") "
Oct 30 01:00:29.485049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.485001    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-kube-api-access-mf7sf\" (UniqueName: \"kubernetes.io/projected/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-kube-api-access-mf7sf\") pod \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") "
Oct 30 01:00:29.485303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.485116    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-openshift-ca\") pod \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\" (UID: \"6187ebe3-e4f8-401f-93a8-e9d60226b8f7\") "
Oct 30 01:00:29.486135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.486105    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-openshift-ca" pod "6187ebe3-e4f8-401f-93a8-e9d60226b8f7" (UID: "6187ebe3-e4f8-401f-93a8-e9d60226b8f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:29.505020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.488862    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-xks-ca" pod "6187ebe3-e4f8-401f-93a8-e9d60226b8f7" (UID: "6187ebe3-e4f8-401f-93a8-e9d60226b8f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:29.505020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.490366    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-kube-api-access-mf7sf" pod "6187ebe3-e4f8-401f-93a8-e9d60226b8f7" (UID: "6187ebe3-e4f8-401f-93a8-e9d60226b8f7"). InnerVolumeSpecName "kube-api-access-mf7sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:29.520662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.520628    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:29.522312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.522288    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:29.525295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.525271    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:00:29.586943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.586912    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:29.586943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.586942    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:29.587195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:29.586954    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf7sf\" (UniqueName: \"kubernetes.io/projected/6187ebe3-e4f8-401f-93a8-e9d60226b8f7-kube-api-access-mf7sf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:30.204629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.204593    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"
Oct 30 01:00:30.204629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.204604    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb" event={"ID":"6187ebe3-e4f8-401f-93a8-e9d60226b8f7","Type":"ContainerDied","Data":"62bfa378ba5208ec5f496923fd0258b229265c80f5d32f87667e9e8058a32f3b"}
Oct 30 01:00:30.204895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.204680    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62bfa378ba5208ec5f496923fd0258b229265c80f5d32f87667e9e8058a32f3b"
Oct 30 01:00:30.688506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.688471    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"]
Oct 30 01:00:30.689311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.689289    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:30.689412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.689313    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:30.689412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.689363    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6187ebe3-e4f8-401f-93a8-e9d60226b8f7" containerName="revoke-keys"
Oct 30 01:00:30.689412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.689372    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6187ebe3-e4f8-401f-93a8-e9d60226b8f7" containerName="revoke-keys"
Oct 30 01:00:30.689623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.689607    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6187ebe3-e4f8-401f-93a8-e9d60226b8f7" containerName="revoke-keys"
Oct 30 01:00:30.689674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.689636    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17f8ea3-9bf4-4ee0-a8a3-74812fe632e9" containerName="cleanup"
Oct 30 01:00:30.693578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.693557    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.710128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.710102    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"]
Oct 30 01:00:30.716259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.716225    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:30.716259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.716226    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:30.716414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.716265    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:30.718628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.718345    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:00:30.729971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.729944    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.730156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.730040    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgx8d\" (UniqueName: \"kubernetes.io/projected/fbb64421-908c-43f1-b1d5-cb81761f5998-kube-api-access-xgx8d\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.730311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.730223    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.831810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.831774    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.831975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.831845    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx8d\" (UniqueName: \"kubernetes.io/projected/fbb64421-908c-43f1-b1d5-cb81761f5998-kube-api-access-xgx8d\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.831975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.831946    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.832626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.832602    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.834233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.834212    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.852430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.852402    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx8d\" (UniqueName: \"kubernetes.io/projected/fbb64421-908c-43f1-b1d5-cb81761f5998-kube-api-access-xgx8d\") pod \"maas-api-revoke-sub-aeacf087-4g2vl\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:30.865625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.865601    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"]
Oct 30 01:00:30.870224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.870207    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:30.889559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.889529    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"]
Oct 30 01:00:30.933427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.933393    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:30.933630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.933595    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:30.933717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:30.933699    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zxd\" (UniqueName: \"kubernetes.io/projected/52e59b88-b95e-4cc5-8507-63901accefb1-kube-api-access-l8zxd\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.009167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.009088    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:31.035078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.035049    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zxd\" (UniqueName: \"kubernetes.io/projected/52e59b88-b95e-4cc5-8507-63901accefb1-kube-api-access-l8zxd\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.035292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.035219    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.035396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.035368    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.036186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.036159    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.037844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.037820    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.053294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.053270    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zxd\" (UniqueName: \"kubernetes.io/projected/52e59b88-b95e-4cc5-8507-63901accefb1-kube-api-access-l8zxd\") pod \"maas-api-revoke-sub-f88485e4-cn6tk\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.185266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.185233    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:31.316798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.316630    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"]
Oct 30 01:00:31.489211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.489183    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"]
Oct 30 01:00:31.489827 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:31.489800    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e59b88_b95e_4cc5_8507_63901accefb1.slice/crio-9e26334f74d6481339b360cb616c19f16eda7deaa4b1a17610bd5025000417e5 WatchSource:0}: Error finding container 9e26334f74d6481339b360cb616c19f16eda7deaa4b1a17610bd5025000417e5: Status 404 returned error can't find the container with id 9e26334f74d6481339b360cb616c19f16eda7deaa4b1a17610bd5025000417e5
Oct 30 01:00:31.717717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.717683    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"]
Oct 30 01:00:31.722078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.722056    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.740034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.739904    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"]
Oct 30 01:00:31.848157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.848119    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.848324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.848243    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6m69\" (UniqueName: \"kubernetes.io/projected/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-kube-api-access-d6m69\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.848437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.848418    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.949805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.949718    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.949805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.949783    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6m69\" (UniqueName: \"kubernetes.io/projected/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-kube-api-access-d6m69\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.950008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.949923    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.950354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.950331    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.952605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.952576    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:31.967928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:31.967901    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6m69\" (UniqueName: \"kubernetes.io/projected/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-kube-api-access-d6m69\") pod \"maas-api-revoke-sub-3cf56dca-64qhl\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:32.037017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.036979    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:32.226897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.226795    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl" event={"ID":"fbb64421-908c-43f1-b1d5-cb81761f5998","Type":"ContainerStarted","Data":"805db9de441ff5c7eac22b8e05898270a8260925f60e40e8b8bc986fffda0df4"}
Oct 30 01:00:32.227075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.226901    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl" event={"ID":"fbb64421-908c-43f1-b1d5-cb81761f5998","Type":"ContainerStarted","Data":"2320337010e5f8c1f38891d6492d5856fcb346e9b9c26af3bda30f408a8a56cf"}
Oct 30 01:00:32.228673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.228646    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk" event={"ID":"52e59b88-b95e-4cc5-8507-63901accefb1","Type":"ContainerStarted","Data":"7208130a5f2bbe9e8f8164835cb928f0a4047b982c3764b0ca7de49712f7ad84"}
Oct 30 01:00:32.228778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.228692    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk" event={"ID":"52e59b88-b95e-4cc5-8507-63901accefb1","Type":"ContainerStarted","Data":"9e26334f74d6481339b360cb616c19f16eda7deaa4b1a17610bd5025000417e5"}
Oct 30 01:00:32.264590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.264534    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl" podStartSLOduration=2.264518083 podStartE2EDuration="2.264518083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:32.261935278 +0000 UTC m=+1073.809625770" watchObservedRunningTime="2026-10-30 01:00:32.264518083 +0000 UTC m=+1073.812208579"
Oct 30 01:00:32.300259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.300199    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk" podStartSLOduration=2.300184978 podStartE2EDuration="2.300184978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:32.297118036 +0000 UTC m=+1073.844808533" watchObservedRunningTime="2026-10-30 01:00:32.300184978 +0000 UTC m=+1073.847875473"
Oct 30 01:00:32.351175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:32.351143    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"]
Oct 30 01:00:32.351676 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:32.351651    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd5844e_6bb5_46f9_84a1_6aa20b91ecc8.slice/crio-db1116cb93750893037163da1ad20bc23f63dc7bcf7c6c8b9518dbb7076e0dc2 WatchSource:0}: Error finding container db1116cb93750893037163da1ad20bc23f63dc7bcf7c6c8b9518dbb7076e0dc2: Status 404 returned error can't find the container with id db1116cb93750893037163da1ad20bc23f63dc7bcf7c6c8b9518dbb7076e0dc2
Oct 30 01:00:33.238981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:33.238944    5104 generic.go:356] "Generic (PLEG): container finished" podID="fbb64421-908c-43f1-b1d5-cb81761f5998" containerID="805db9de441ff5c7eac22b8e05898270a8260925f60e40e8b8bc986fffda0df4" exitCode=0
Oct 30 01:00:33.239429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:33.239031    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl" event={"ID":"fbb64421-908c-43f1-b1d5-cb81761f5998","Type":"ContainerDied","Data":"805db9de441ff5c7eac22b8e05898270a8260925f60e40e8b8bc986fffda0df4"}
Oct 30 01:00:33.240902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:33.240879    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl" event={"ID":"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8","Type":"ContainerStarted","Data":"55670830ea1d7afb1653e5f68b677d71200dc2b5d9251a9ffed312b9d12860a4"}
Oct 30 01:00:33.240986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:33.240927    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl" event={"ID":"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8","Type":"ContainerStarted","Data":"db1116cb93750893037163da1ad20bc23f63dc7bcf7c6c8b9518dbb7076e0dc2"}
Oct 30 01:00:33.242643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:33.242621    5104 generic.go:356] "Generic (PLEG): container finished" podID="52e59b88-b95e-4cc5-8507-63901accefb1" containerID="7208130a5f2bbe9e8f8164835cb928f0a4047b982c3764b0ca7de49712f7ad84" exitCode=0
Oct 30 01:00:33.242740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:33.242667    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk" event={"ID":"52e59b88-b95e-4cc5-8507-63901accefb1","Type":"ContainerDied","Data":"7208130a5f2bbe9e8f8164835cb928f0a4047b982c3764b0ca7de49712f7ad84"}
Oct 30 01:00:33.314039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:33.313986    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl" podStartSLOduration=2.313971992 podStartE2EDuration="2.313971992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:33.309767448 +0000 UTC m=+1074.857457946" watchObservedRunningTime="2026-10-30 01:00:33.313971992 +0000 UTC m=+1074.861662521"
Oct 30 01:00:34.253314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.253277    5104 generic.go:356] "Generic (PLEG): container finished" podID="9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" containerID="55670830ea1d7afb1653e5f68b677d71200dc2b5d9251a9ffed312b9d12860a4" exitCode=0
Oct 30 01:00:34.253760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.253364    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl" event={"ID":"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8","Type":"ContainerDied","Data":"55670830ea1d7afb1653e5f68b677d71200dc2b5d9251a9ffed312b9d12860a4"}
Oct 30 01:00:34.467790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.467764    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:34.517813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.517746    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:34.591213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.591179    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-xks-ca\") pod \"fbb64421-908c-43f1-b1d5-cb81761f5998\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") "
Oct 30 01:00:34.591556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.591536    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb64421-908c-43f1-b1d5-cb81761f5998-kube-api-access-xgx8d\" (UniqueName: \"kubernetes.io/projected/fbb64421-908c-43f1-b1d5-cb81761f5998-kube-api-access-xgx8d\") pod \"fbb64421-908c-43f1-b1d5-cb81761f5998\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") "
Oct 30 01:00:34.591802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.591788    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-openshift-ca\") pod \"fbb64421-908c-43f1-b1d5-cb81761f5998\" (UID: \"fbb64421-908c-43f1-b1d5-cb81761f5998\") "
Oct 30 01:00:34.591949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.591933    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e59b88-b95e-4cc5-8507-63901accefb1-kube-api-access-l8zxd\" (UniqueName: \"kubernetes.io/projected/52e59b88-b95e-4cc5-8507-63901accefb1-kube-api-access-l8zxd\") pod \"52e59b88-b95e-4cc5-8507-63901accefb1\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") "
Oct 30 01:00:34.592077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.592064    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-openshift-ca\") pod \"52e59b88-b95e-4cc5-8507-63901accefb1\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") "
Oct 30 01:00:34.592194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.592182    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-xks-ca\") pod \"52e59b88-b95e-4cc5-8507-63901accefb1\" (UID: \"52e59b88-b95e-4cc5-8507-63901accefb1\") "
Oct 30 01:00:34.593719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.593685    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-openshift-ca" pod "52e59b88-b95e-4cc5-8507-63901accefb1" (UID: "52e59b88-b95e-4cc5-8507-63901accefb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:34.594419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.594391    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-openshift-ca" pod "fbb64421-908c-43f1-b1d5-cb81761f5998" (UID: "fbb64421-908c-43f1-b1d5-cb81761f5998"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:34.607840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.607773    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e59b88-b95e-4cc5-8507-63901accefb1-kube-api-access-l8zxd" pod "52e59b88-b95e-4cc5-8507-63901accefb1" (UID: "52e59b88-b95e-4cc5-8507-63901accefb1"). InnerVolumeSpecName "kube-api-access-l8zxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:34.607840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.607800    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-xks-ca" pod "fbb64421-908c-43f1-b1d5-cb81761f5998" (UID: "fbb64421-908c-43f1-b1d5-cb81761f5998"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:34.607995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.607916    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-xks-ca" pod "52e59b88-b95e-4cc5-8507-63901accefb1" (UID: "52e59b88-b95e-4cc5-8507-63901accefb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:34.609575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.609534    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb64421-908c-43f1-b1d5-cb81761f5998-kube-api-access-xgx8d" pod "fbb64421-908c-43f1-b1d5-cb81761f5998" (UID: "fbb64421-908c-43f1-b1d5-cb81761f5998"). InnerVolumeSpecName "kube-api-access-xgx8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:34.694382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.694347    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgx8d\" (UniqueName: \"kubernetes.io/projected/fbb64421-908c-43f1-b1d5-cb81761f5998-kube-api-access-xgx8d\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:34.694382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.694377    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:34.694592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.694405    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8zxd\" (UniqueName: \"kubernetes.io/projected/52e59b88-b95e-4cc5-8507-63901accefb1-kube-api-access-l8zxd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:34.694592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.694418    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:34.694592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.694432    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e59b88-b95e-4cc5-8507-63901accefb1-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:34.694592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:34.694443    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb64421-908c-43f1-b1d5-cb81761f5998-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:35.272875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.272845    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"
Oct 30 01:00:35.273341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.272880    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl" event={"ID":"fbb64421-908c-43f1-b1d5-cb81761f5998","Type":"ContainerDied","Data":"2320337010e5f8c1f38891d6492d5856fcb346e9b9c26af3bda30f408a8a56cf"}
Oct 30 01:00:35.273341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.272973    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2320337010e5f8c1f38891d6492d5856fcb346e9b9c26af3bda30f408a8a56cf"
Oct 30 01:00:35.275523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.275483    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk" event={"ID":"52e59b88-b95e-4cc5-8507-63901accefb1","Type":"ContainerDied","Data":"9e26334f74d6481339b360cb616c19f16eda7deaa4b1a17610bd5025000417e5"}
Oct 30 01:00:35.275523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.275517    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"
Oct 30 01:00:35.275727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.275535    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e26334f74d6481339b360cb616c19f16eda7deaa4b1a17610bd5025000417e5"
Oct 30 01:00:35.448568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.448542    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:35.506550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.506510    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-openshift-ca\") pod \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") "
Oct 30 01:00:35.506789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.506767    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-xks-ca\") pod \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") "
Oct 30 01:00:35.506965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.506898    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-kube-api-access-d6m69\" (UniqueName: \"kubernetes.io/projected/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-kube-api-access-d6m69\") pod \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\" (UID: \"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8\") "
Oct 30 01:00:35.507367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.507342    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-openshift-ca" pod "9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" (UID: "9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:35.510503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.509488    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-xks-ca" pod "9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" (UID: "9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:35.515615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.515583    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-kube-api-access-d6m69" pod "9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" (UID: "9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8"). InnerVolumeSpecName "kube-api-access-d6m69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:35.609819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.609719    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:35.609819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.609758    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6m69\" (UniqueName: \"kubernetes.io/projected/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-kube-api-access-d6m69\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:35.609819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:35.609777    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:36.288517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.288483    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"
Oct 30 01:00:36.288517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.288506    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl" event={"ID":"9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8","Type":"ContainerDied","Data":"db1116cb93750893037163da1ad20bc23f63dc7bcf7c6c8b9518dbb7076e0dc2"}
Oct 30 01:00:36.289108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.288562    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db1116cb93750893037163da1ad20bc23f63dc7bcf7c6c8b9518dbb7076e0dc2"
Oct 30 01:00:36.781981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.781944    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"]
Oct 30 01:00:36.782947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.782924    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fbb64421-908c-43f1-b1d5-cb81761f5998" containerName="revoke-keys"
Oct 30 01:00:36.783031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.782952    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb64421-908c-43f1-b1d5-cb81761f5998" containerName="revoke-keys"
Oct 30 01:00:36.783031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.782992    5104 cpu_manager.go:414] "containerMap: removing container" podUID="52e59b88-b95e-4cc5-8507-63901accefb1" containerName="revoke-keys"
Oct 30 01:00:36.783031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.783001    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e59b88-b95e-4cc5-8507-63901accefb1" containerName="revoke-keys"
Oct 30 01:00:36.783137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.783038    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" containerName="revoke-keys"
Oct 30 01:00:36.783137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.783047    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" containerName="revoke-keys"
Oct 30 01:00:36.783253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.783235    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" containerName="revoke-keys"
Oct 30 01:00:36.783338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.783264    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb64421-908c-43f1-b1d5-cb81761f5998" containerName="revoke-keys"
Oct 30 01:00:36.783338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.783278    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e59b88-b95e-4cc5-8507-63901accefb1" containerName="revoke-keys"
Oct 30 01:00:36.787987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.787956    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.820153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.820122    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"]
Oct 30 01:00:36.931563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.931513    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.931761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.931665    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.931761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.931735    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.931900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.931845    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.931964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.931901    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.931964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.931920    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.932070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.931984    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlq7d\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-kube-api-access-zlq7d\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.932149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.932130    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:36.932269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:36.932174    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033440    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033496    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033556    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033605    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033624    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033684    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033740    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033767    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033825    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlq7d\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-kube-api-access-zlq7d\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.033917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033887    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.034116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.033954    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.034170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.034112    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.034217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.034194    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.034355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.034331    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.036051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.036026    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.036367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.036350    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.051862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.051834    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.052320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.052297    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlq7d\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-kube-api-access-zlq7d\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:37.103922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:37.103881    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:38.004078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:38.004037    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"]
Oct 30 01:00:38.160598 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:38.156662    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa02a4f_e4f4_4b34_b165_e306f42c7c2a.slice/crio-c4684e29a0b46f28da7f828bea7d817257623c46e009f178486f6b0342f95d40 WatchSource:0}: Error finding container c4684e29a0b46f28da7f828bea7d817257623c46e009f178486f6b0342f95d40: Status 404 returned error can't find the container with id c4684e29a0b46f28da7f828bea7d817257623c46e009f178486f6b0342f95d40
Oct 30 01:00:38.162324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:38.160961    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:38.162324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:38.161070    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:38.162324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:38.161113    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:00:38.434166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:38.434127    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" event={"ID":"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a","Type":"ContainerStarted","Data":"c4684e29a0b46f28da7f828bea7d817257623c46e009f178486f6b0342f95d40"}
Oct 30 01:00:38.804493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:38.773730    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.971653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:38.971600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:39.006915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.006871    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:39.477890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.477752    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:39.510903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.510861    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:39.646058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.645980    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:00:39.773548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.770757    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:39.858521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.848980    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" event={"ID":"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a","Type":"ContainerStarted","Data":"ef71142048cbf7d798d5500d173d1f669985f8f84ba59d33f9a181ea6e674cc4"}
Oct 30 01:00:39.931558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.931438    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podStartSLOduration=3.93141942 podStartE2EDuration="3.93141942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:39.90740178 +0000 UTC m=+1081.455092278" watchObservedRunningTime="2026-10-30 01:00:39.93141942 +0000 UTC m=+1081.479109916"
Oct 30 01:00:39.957206 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:00:39.957169    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:00:40.005534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.976963    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:40.005534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:39.977728    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:00:40.125236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.125200    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:40.125941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.125913    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:40.126115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.126087    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:40.243859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.243815    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:40.270573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.260088    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:00:40.639538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.482898    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:00:40.688912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.518980    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:40.689153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.536853    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:40.716582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.715542    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:40.716582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.716280    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:00:40.736526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.727566    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:40.771273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:40.771229    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:41.076992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.051557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:41.121529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.121443    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:41.121835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.121783    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:41.266815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.266759    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.267772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.267741    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:00:41.310488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.288044    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:00:41.518486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.493706    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:00:41.518830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.518803    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:00:41.518984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.518971    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:00:41.545619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.545584    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:00:41.545911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.545892    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:00:41.618480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.616417    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:00:41.730481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.729557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.759493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.756833    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:00:41.759493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.757778    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.801552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.789535    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:00:41.809486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.801787    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.829109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.828975    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.832811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.829420    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:00:41.833161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:41.833141    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:00:42.056345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.052651    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:42.158490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.158276    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:42.158490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.158332    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:42.274391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.274347    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:42.478714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.328074    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:00:42.478714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.437276    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:42.478714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.437345    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:42.879547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.779137    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:42.879547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.799570    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:42.879547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.808959    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:42.879547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.811070    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:42.963056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.962999    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:00:42.976496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.965811    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 01:00:42.976876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:42.976838    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:43.072510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.054160    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:43.106192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.106157    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:43.106489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.106443    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:43.453505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.448114    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:43.453505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.449109    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:00:43.779706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.779596    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:43.809662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.809607    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:43.816342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.809992    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:43.816923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.813225    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:43.818601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.818579    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:00:43.974746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:43.967585    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:44.002631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.002568    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:44.115313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.113595    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:44.115313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.114938    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:44.115313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.114983    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:44.448698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.448629    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:44.718084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.715670    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:00:44.789547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.787596    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:44.817054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.816227    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:00:44.821843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.820182    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:00:44.821843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.820517    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:00:44.874260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.859615    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:44.970965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:44.969652    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:45.007776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.007731    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:00:45.109740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.109212    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:45.109740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.109282    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:45.114577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.113798    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:45.370779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.370681    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:00:45.442885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.442843    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:00:45.554630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.554397    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:00:45.570567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.570519    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:00:45.572177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.572148    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:00:45.572392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.572377    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:00:45.641287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:45.641199    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:00:46.113482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.111256    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:46.113482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.111321    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:46.170894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.170577    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"]
Oct 30 01:00:46.192508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.182087    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"]
Oct 30 01:00:46.192508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.182208    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.202483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.201010    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:46.202483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.201086    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:46.206548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.203812    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:00:46.231350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.231315    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ndh\" (UniqueName: \"kubernetes.io/projected/6c962e72-cff0-4575-b4b5-beabdfecef5c-kube-api-access-76ndh\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.231842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.231813    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.231954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.231931    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.303789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.303750    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:46.333271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.333233    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.333484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.333380    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ndh\" (UniqueName: \"kubernetes.io/projected/6c962e72-cff0-4575-b4b5-beabdfecef5c-kube-api-access-76ndh\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.333593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.333576    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.333919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.333900    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.336023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.335998    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.351679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.351654    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ndh\" (UniqueName: \"kubernetes.io/projected/6c962e72-cff0-4575-b4b5-beabdfecef5c-kube-api-access-76ndh\") pod \"maas-api-revoke-sub-506b3861-j25df\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.538178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.538148    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:46.715423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.715378    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"]
Oct 30 01:00:46.721995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.721970    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.735360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.735320    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"]
Oct 30 01:00:46.840984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.840896    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwz87\" (UniqueName: \"kubernetes.io/projected/a52c6cdc-e6de-4f33-871f-52e4b00e6494-kube-api-access-pwz87\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.842255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.842221    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.842642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.842623    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.912656 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:46.912614    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c962e72_cff0_4575_b4b5_beabdfecef5c.slice/crio-3970bf91bbbf3572e3344cb268197233f7ddc917a88d2d08de14ecc919f85369 WatchSource:0}: Error finding container 3970bf91bbbf3572e3344cb268197233f7ddc917a88d2d08de14ecc919f85369: Status 404 returned error can't find the container with id 3970bf91bbbf3572e3344cb268197233f7ddc917a88d2d08de14ecc919f85369
Oct 30 01:00:46.915397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.915371    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"]
Oct 30 01:00:46.945872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.945827    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwz87\" (UniqueName: \"kubernetes.io/projected/a52c6cdc-e6de-4f33-871f-52e4b00e6494-kube-api-access-pwz87\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.946019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.945937    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.946019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.945965    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.946777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.946752    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.948300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.948276    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:46.963817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:46.963791    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwz87\" (UniqueName: \"kubernetes.io/projected/a52c6cdc-e6de-4f33-871f-52e4b00e6494-kube-api-access-pwz87\") pod \"maas-api-revoke-sub-c751398f-spq5f\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:47.062431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.062389    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:47.109886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.109219    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:47.109886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.109294    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:47.408172 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:47.408021    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52c6cdc_e6de_4f33_871f_52e4b00e6494.slice/crio-b132295c41131cbc8d4fd469a03f0ab96076df84e26264251ffa34e87cdfe832 WatchSource:0}: Error finding container b132295c41131cbc8d4fd469a03f0ab96076df84e26264251ffa34e87cdfe832: Status 404 returned error can't find the container with id b132295c41131cbc8d4fd469a03f0ab96076df84e26264251ffa34e87cdfe832
Oct 30 01:00:47.412313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.411709    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"]
Oct 30 01:00:47.702596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.701270    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df" event={"ID":"6c962e72-cff0-4575-b4b5-beabdfecef5c","Type":"ContainerStarted","Data":"d40ec4da66a41953bbad1654716cee894cac82a2fef2d5608700a7f0b1b3d1f1"}
Oct 30 01:00:47.702596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.701354    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df" event={"ID":"6c962e72-cff0-4575-b4b5-beabdfecef5c","Type":"ContainerStarted","Data":"3970bf91bbbf3572e3344cb268197233f7ddc917a88d2d08de14ecc919f85369"}
Oct 30 01:00:47.716546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.716421    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f" event={"ID":"a52c6cdc-e6de-4f33-871f-52e4b00e6494","Type":"ContainerStarted","Data":"b132295c41131cbc8d4fd469a03f0ab96076df84e26264251ffa34e87cdfe832"}
Oct 30 01:00:47.756171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.756101    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df" podStartSLOduration=1.7560784809999999 podStartE2EDuration="1.756078481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:47.747806985 +0000 UTC m=+1089.295497483" watchObservedRunningTime="2026-10-30 01:00:47.756078481 +0000 UTC m=+1089.303768978"
Oct 30 01:00:47.771859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.771822    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"]
Oct 30 01:00:47.831721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.830961    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"]
Oct 30 01:00:47.831942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.831761    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.862871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.862844    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg75m\" (UniqueName: \"kubernetes.io/projected/1591081c-a77d-455b-9ba2-b01dc213fa47-kube-api-access-lg75m\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.863076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.863061    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.863303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.863275    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.966149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.966057    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg75m\" (UniqueName: \"kubernetes.io/projected/1591081c-a77d-455b-9ba2-b01dc213fa47-kube-api-access-lg75m\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.966149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.966105    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.966394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.966193    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.967057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.967027    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:47.968678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:47.968656    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:48.007346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.007309    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg75m\" (UniqueName: \"kubernetes.io/projected/1591081c-a77d-455b-9ba2-b01dc213fa47-kube-api-access-lg75m\") pod \"maas-api-revoke-sub-eea7cd55-zvwpm\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:48.105534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.105442    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:48.105693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.105526    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:48.154903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.154866    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:48.503794 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:48.503763    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1591081c_a77d_455b_9ba2_b01dc213fa47.slice/crio-cdaf8c8948710ba339df7a959daa15ac1ab1804a73d33f8d7ab63b5606784272 WatchSource:0}: Error finding container cdaf8c8948710ba339df7a959daa15ac1ab1804a73d33f8d7ab63b5606784272: Status 404 returned error can't find the container with id cdaf8c8948710ba339df7a959daa15ac1ab1804a73d33f8d7ab63b5606784272
Oct 30 01:00:48.505782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.505749    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"]
Oct 30 01:00:48.726596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.726562    5104 generic.go:356] "Generic (PLEG): container finished" podID="6c962e72-cff0-4575-b4b5-beabdfecef5c" containerID="d40ec4da66a41953bbad1654716cee894cac82a2fef2d5608700a7f0b1b3d1f1" exitCode=0
Oct 30 01:00:48.726773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.726641    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df" event={"ID":"6c962e72-cff0-4575-b4b5-beabdfecef5c","Type":"ContainerDied","Data":"d40ec4da66a41953bbad1654716cee894cac82a2fef2d5608700a7f0b1b3d1f1"}
Oct 30 01:00:48.728612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.728585    5104 generic.go:356] "Generic (PLEG): container finished" podID="a52c6cdc-e6de-4f33-871f-52e4b00e6494" containerID="f1fc7709afe9c71f3773e3768b13511ab3908dd280d62d813c6bdbf77699e7e4" exitCode=0
Oct 30 01:00:48.728756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.728667    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f" event={"ID":"a52c6cdc-e6de-4f33-871f-52e4b00e6494","Type":"ContainerDied","Data":"f1fc7709afe9c71f3773e3768b13511ab3908dd280d62d813c6bdbf77699e7e4"}
Oct 30 01:00:48.729996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:48.729975    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm" event={"ID":"1591081c-a77d-455b-9ba2-b01dc213fa47","Type":"ContainerStarted","Data":"cdaf8c8948710ba339df7a959daa15ac1ab1804a73d33f8d7ab63b5606784272"}
Oct 30 01:00:49.104642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:49.104542    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:49.104642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:49.104599    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:49.740747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:49.740711    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm" event={"ID":"1591081c-a77d-455b-9ba2-b01dc213fa47","Type":"ContainerStarted","Data":"6fcf931b15c0154d7104038fbd9ec87d22804518ba3aa41ffd6f513cbaa2ff80"}
Oct 30 01:00:49.779656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:49.778308    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm" podStartSLOduration=2.778288827 podStartE2EDuration="2.778288827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:49.777858119 +0000 UTC m=+1091.325548618" watchObservedRunningTime="2026-10-30 01:00:49.778288827 +0000 UTC m=+1091.325979324"
Oct 30 01:00:49.981524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:49.972049    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:50.031308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.031267    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:50.074475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.074225    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-xks-ca\") pod \"6c962e72-cff0-4575-b4b5-beabdfecef5c\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") "
Oct 30 01:00:50.074475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.074394    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52c6cdc-e6de-4f33-871f-52e4b00e6494-kube-api-access-pwz87\" (UniqueName: \"kubernetes.io/projected/a52c6cdc-e6de-4f33-871f-52e4b00e6494-kube-api-access-pwz87\") pod \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") "
Oct 30 01:00:50.074475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.074429    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-openshift-ca\") pod \"6c962e72-cff0-4575-b4b5-beabdfecef5c\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") "
Oct 30 01:00:50.074775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.074507    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-xks-ca\") pod \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") "
Oct 30 01:00:50.074775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.074566    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c962e72-cff0-4575-b4b5-beabdfecef5c-kube-api-access-76ndh\" (UniqueName: \"kubernetes.io/projected/6c962e72-cff0-4575-b4b5-beabdfecef5c-kube-api-access-76ndh\") pod \"6c962e72-cff0-4575-b4b5-beabdfecef5c\" (UID: \"6c962e72-cff0-4575-b4b5-beabdfecef5c\") "
Oct 30 01:00:50.074775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.074709    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-openshift-ca\") pod \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\" (UID: \"a52c6cdc-e6de-4f33-871f-52e4b00e6494\") "
Oct 30 01:00:50.076038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.075745    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-openshift-ca" pod "a52c6cdc-e6de-4f33-871f-52e4b00e6494" (UID: "a52c6cdc-e6de-4f33-871f-52e4b00e6494"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:50.077908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.077882    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-openshift-ca" pod "6c962e72-cff0-4575-b4b5-beabdfecef5c" (UID: "6c962e72-cff0-4575-b4b5-beabdfecef5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:50.080661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.080625    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c962e72-cff0-4575-b4b5-beabdfecef5c-kube-api-access-76ndh" pod "6c962e72-cff0-4575-b4b5-beabdfecef5c" (UID: "6c962e72-cff0-4575-b4b5-beabdfecef5c"). InnerVolumeSpecName "kube-api-access-76ndh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:50.082925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.082900    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-xks-ca" pod "a52c6cdc-e6de-4f33-871f-52e4b00e6494" (UID: "a52c6cdc-e6de-4f33-871f-52e4b00e6494"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:50.083337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.083316    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-xks-ca" pod "6c962e72-cff0-4575-b4b5-beabdfecef5c" (UID: "6c962e72-cff0-4575-b4b5-beabdfecef5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:50.083719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.083700    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52c6cdc-e6de-4f33-871f-52e4b00e6494-kube-api-access-pwz87" pod "a52c6cdc-e6de-4f33-871f-52e4b00e6494" (UID: "a52c6cdc-e6de-4f33-871f-52e4b00e6494"). InnerVolumeSpecName "kube-api-access-pwz87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:50.107236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.107168    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:00:50.107886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.107482    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:00:50.177369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.177324    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwz87\" (UniqueName: \"kubernetes.io/projected/a52c6cdc-e6de-4f33-871f-52e4b00e6494-kube-api-access-pwz87\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:50.177369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.177364    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:50.177669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.177384    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:50.177669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.177399    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ndh\" (UniqueName: \"kubernetes.io/projected/6c962e72-cff0-4575-b4b5-beabdfecef5c-kube-api-access-76ndh\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:50.177669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.177415    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52c6cdc-e6de-4f33-871f-52e4b00e6494-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:50.177669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.177432    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c962e72-cff0-4575-b4b5-beabdfecef5c-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:50.753048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.753013    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"
Oct 30 01:00:50.753048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.753025    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df" event={"ID":"6c962e72-cff0-4575-b4b5-beabdfecef5c","Type":"ContainerDied","Data":"3970bf91bbbf3572e3344cb268197233f7ddc917a88d2d08de14ecc919f85369"}
Oct 30 01:00:50.753621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.753078    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3970bf91bbbf3572e3344cb268197233f7ddc917a88d2d08de14ecc919f85369"
Oct 30 01:00:50.754918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.754897    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"
Oct 30 01:00:50.755052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.754940    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f" event={"ID":"a52c6cdc-e6de-4f33-871f-52e4b00e6494","Type":"ContainerDied","Data":"b132295c41131cbc8d4fd469a03f0ab96076df84e26264251ffa34e87cdfe832"}
Oct 30 01:00:50.755052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.755021    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b132295c41131cbc8d4fd469a03f0ab96076df84e26264251ffa34e87cdfe832"
Oct 30 01:00:50.757651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.757625    5104 generic.go:356] "Generic (PLEG): container finished" podID="1591081c-a77d-455b-9ba2-b01dc213fa47" containerID="6fcf931b15c0154d7104038fbd9ec87d22804518ba3aa41ffd6f513cbaa2ff80" exitCode=0
Oct 30 01:00:50.757782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:50.757731    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm" event={"ID":"1591081c-a77d-455b-9ba2-b01dc213fa47","Type":"ContainerDied","Data":"6fcf931b15c0154d7104038fbd9ec87d22804518ba3aa41ffd6f513cbaa2ff80"}
Oct 30 01:00:51.383376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.383339    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:51.383854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.383831    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:51.385971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.385947    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:00:51.963328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.963305    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:51.995470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.995427    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-openshift-ca\") pod \"1591081c-a77d-455b-9ba2-b01dc213fa47\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") "
Oct 30 01:00:51.995679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.995501    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-xks-ca\") pod \"1591081c-a77d-455b-9ba2-b01dc213fa47\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") "
Oct 30 01:00:51.995834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.995812    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-openshift-ca" pod "1591081c-a77d-455b-9ba2-b01dc213fa47" (UID: "1591081c-a77d-455b-9ba2-b01dc213fa47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:51.996098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.996069    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:51.997537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:51.997515    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-xks-ca" pod "1591081c-a77d-455b-9ba2-b01dc213fa47" (UID: "1591081c-a77d-455b-9ba2-b01dc213fa47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:52.096809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:52.096712    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1591081c-a77d-455b-9ba2-b01dc213fa47-kube-api-access-lg75m\" (UniqueName: \"kubernetes.io/projected/1591081c-a77d-455b-9ba2-b01dc213fa47-kube-api-access-lg75m\") pod \"1591081c-a77d-455b-9ba2-b01dc213fa47\" (UID: \"1591081c-a77d-455b-9ba2-b01dc213fa47\") "
Oct 30 01:00:52.097193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:52.097165    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1591081c-a77d-455b-9ba2-b01dc213fa47-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:52.098989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:52.098960    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1591081c-a77d-455b-9ba2-b01dc213fa47-kube-api-access-lg75m" pod "1591081c-a77d-455b-9ba2-b01dc213fa47" (UID: "1591081c-a77d-455b-9ba2-b01dc213fa47"). InnerVolumeSpecName "kube-api-access-lg75m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:52.198664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:52.198625    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg75m\" (UniqueName: \"kubernetes.io/projected/1591081c-a77d-455b-9ba2-b01dc213fa47-kube-api-access-lg75m\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:52.790277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:52.790249    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"
Oct 30 01:00:52.790643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:52.790612    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm" event={"ID":"1591081c-a77d-455b-9ba2-b01dc213fa47","Type":"ContainerDied","Data":"cdaf8c8948710ba339df7a959daa15ac1ab1804a73d33f8d7ab63b5606784272"}
Oct 30 01:00:52.790776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:52.790750    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdaf8c8948710ba339df7a959daa15ac1ab1804a73d33f8d7ab63b5606784272"
Oct 30 01:00:53.635224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.635186    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"]
Oct 30 01:00:53.635960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.635942    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1591081c-a77d-455b-9ba2-b01dc213fa47" containerName="revoke-keys"
Oct 30 01:00:53.635960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.635960    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1591081c-a77d-455b-9ba2-b01dc213fa47" containerName="revoke-keys"
Oct 30 01:00:53.636064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.636003    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6c962e72-cff0-4575-b4b5-beabdfecef5c" containerName="revoke-keys"
Oct 30 01:00:53.636064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.636008    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c962e72-cff0-4575-b4b5-beabdfecef5c" containerName="revoke-keys"
Oct 30 01:00:53.636064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.636017    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a52c6cdc-e6de-4f33-871f-52e4b00e6494" containerName="revoke-keys"
Oct 30 01:00:53.636064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.636023    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52c6cdc-e6de-4f33-871f-52e4b00e6494" containerName="revoke-keys"
Oct 30 01:00:53.636213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.636118    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c962e72-cff0-4575-b4b5-beabdfecef5c" containerName="revoke-keys"
Oct 30 01:00:53.636213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.636127    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52c6cdc-e6de-4f33-871f-52e4b00e6494" containerName="revoke-keys"
Oct 30 01:00:53.636213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.636137    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1591081c-a77d-455b-9ba2-b01dc213fa47" containerName="revoke-keys"
Oct 30 01:00:53.641378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.641357    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.654914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.654882    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:53.655387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.655307    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:53.655387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.655376    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:53.658036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.658005    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"]
Oct 30 01:00:53.662839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.662813    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:00:53.719223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.719178    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sv67\" (UniqueName: \"kubernetes.io/projected/cd359f6a-f6c5-48f2-a822-5221332bb572-kube-api-access-6sv67\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.719487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.719441    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.719619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.719539    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.820479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.820438    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.820718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.820697    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.820896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.820881    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sv67\" (UniqueName: \"kubernetes.io/projected/cd359f6a-f6c5-48f2-a822-5221332bb572-kube-api-access-6sv67\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.821604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.821574    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.829375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.829346    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.839512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.839482    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sv67\" (UniqueName: \"kubernetes.io/projected/cd359f6a-f6c5-48f2-a822-5221332bb572-kube-api-access-6sv67\") pod \"maas-api-revoke-sub-760dfd7e-d59p4\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:53.956327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:53.956293    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:54.299516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:54.299434    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"]
Oct 30 01:00:54.324617 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:54.324584    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd359f6a_f6c5_48f2_a822_5221332bb572.slice/crio-7418aade3612aac8022934829529934f83ea9e2d6aba6454430f26c30b7e25da WatchSource:0}: Error finding container 7418aade3612aac8022934829529934f83ea9e2d6aba6454430f26c30b7e25da: Status 404 returned error can't find the container with id 7418aade3612aac8022934829529934f83ea9e2d6aba6454430f26c30b7e25da
Oct 30 01:00:54.810597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:54.810550    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4" event={"ID":"cd359f6a-f6c5-48f2-a822-5221332bb572","Type":"ContainerStarted","Data":"fecb6bd902ae6017e72ab2b175198635770dd4c0511a5a9a3d9816d26a0aa1d6"}
Oct 30 01:00:54.811018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:54.810618    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4" event={"ID":"cd359f6a-f6c5-48f2-a822-5221332bb572","Type":"ContainerStarted","Data":"7418aade3612aac8022934829529934f83ea9e2d6aba6454430f26c30b7e25da"}
Oct 30 01:00:54.846743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:54.846695    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4" podStartSLOduration=1.8466778430000002 podStartE2EDuration="1.846677843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:54.846510618 +0000 UTC m=+1096.394201115" watchObservedRunningTime="2026-10-30 01:00:54.846677843 +0000 UTC m=+1096.394368340"
Oct 30 01:00:55.827428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:55.827385    5104 generic.go:356] "Generic (PLEG): container finished" podID="cd359f6a-f6c5-48f2-a822-5221332bb572" containerID="fecb6bd902ae6017e72ab2b175198635770dd4c0511a5a9a3d9816d26a0aa1d6" exitCode=0
Oct 30 01:00:55.827868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:55.827548    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4" event={"ID":"cd359f6a-f6c5-48f2-a822-5221332bb572","Type":"ContainerDied","Data":"fecb6bd902ae6017e72ab2b175198635770dd4c0511a5a9a3d9816d26a0aa1d6"}
Oct 30 01:00:56.435656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.435621    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"]
Oct 30 01:00:56.451765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.451736    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.462734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.462702    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"]
Oct 30 01:00:56.588450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.588410    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctscg\" (UniqueName: \"kubernetes.io/projected/df0e75a3-027a-41f2-9e15-905c617efcd1-kube-api-access-ctscg\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.588665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.588535    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.588665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.588586    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.690120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.690035    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctscg\" (UniqueName: \"kubernetes.io/projected/df0e75a3-027a-41f2-9e15-905c617efcd1-kube-api-access-ctscg\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.690120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.690100    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.690340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.690143    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.690730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.690709    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.692671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.692649    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.708151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.708122    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctscg\" (UniqueName: \"kubernetes.io/projected/df0e75a3-027a-41f2-9e15-905c617efcd1-kube-api-access-ctscg\") pod \"maas-api-revoke-sub-1cbddc36-ss8zw\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:56.768788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:56.768752    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:00:57.056880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.056849    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:57.085589 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:00:57.085558    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0e75a3_027a_41f2_9e15_905c617efcd1.slice/crio-d50352f1d45e21d835fc168b2e196b15cbce2fd7604245baf34d1fe2c7009409 WatchSource:0}: Error finding container d50352f1d45e21d835fc168b2e196b15cbce2fd7604245baf34d1fe2c7009409: Status 404 returned error can't find the container with id d50352f1d45e21d835fc168b2e196b15cbce2fd7604245baf34d1fe2c7009409
Oct 30 01:00:57.090209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.090179    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"]
Oct 30 01:00:57.095858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.095754    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-openshift-ca\") pod \"cd359f6a-f6c5-48f2-a822-5221332bb572\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") "
Oct 30 01:00:57.095858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.095816    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd359f6a-f6c5-48f2-a822-5221332bb572-kube-api-access-6sv67\" (UniqueName: \"kubernetes.io/projected/cd359f6a-f6c5-48f2-a822-5221332bb572-kube-api-access-6sv67\") pod \"cd359f6a-f6c5-48f2-a822-5221332bb572\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") "
Oct 30 01:00:57.096894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.096133    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-xks-ca\") pod \"cd359f6a-f6c5-48f2-a822-5221332bb572\" (UID: \"cd359f6a-f6c5-48f2-a822-5221332bb572\") "
Oct 30 01:00:57.097700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.097663    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-openshift-ca" pod "cd359f6a-f6c5-48f2-a822-5221332bb572" (UID: "cd359f6a-f6c5-48f2-a822-5221332bb572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:57.100003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.099962    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-xks-ca" pod "cd359f6a-f6c5-48f2-a822-5221332bb572" (UID: "cd359f6a-f6c5-48f2-a822-5221332bb572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:57.101668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.101645    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd359f6a-f6c5-48f2-a822-5221332bb572-kube-api-access-6sv67" pod "cd359f6a-f6c5-48f2-a822-5221332bb572" (UID: "cd359f6a-f6c5-48f2-a822-5221332bb572"). InnerVolumeSpecName "kube-api-access-6sv67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:57.199026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.198992    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:57.199026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.199024    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sv67\" (UniqueName: \"kubernetes.io/projected/cd359f6a-f6c5-48f2-a822-5221332bb572-kube-api-access-6sv67\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:57.199224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.199037    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd359f6a-f6c5-48f2-a822-5221332bb572-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:00:57.848511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.848447    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"
Oct 30 01:00:57.848511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.848489    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4" event={"ID":"cd359f6a-f6c5-48f2-a822-5221332bb572","Type":"ContainerDied","Data":"7418aade3612aac8022934829529934f83ea9e2d6aba6454430f26c30b7e25da"}
Oct 30 01:00:57.848754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.848543    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7418aade3612aac8022934829529934f83ea9e2d6aba6454430f26c30b7e25da"
Oct 30 01:00:57.850319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.850292    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw" event={"ID":"df0e75a3-027a-41f2-9e15-905c617efcd1","Type":"ContainerStarted","Data":"4be6d78ded1528d17a946da07d374205e4d6674d1009329b5133295c0c34544f"}
Oct 30 01:00:57.850423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.850345    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw" event={"ID":"df0e75a3-027a-41f2-9e15-905c617efcd1","Type":"ContainerStarted","Data":"d50352f1d45e21d835fc168b2e196b15cbce2fd7604245baf34d1fe2c7009409"}
Oct 30 01:00:57.894516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:57.894446    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw" podStartSLOduration=1.894426895 podStartE2EDuration="1.894426895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:57.885321142 +0000 UTC m=+1099.433011639" watchObservedRunningTime="2026-10-30 01:00:57.894426895 +0000 UTC m=+1099.442117391"
Oct 30 01:00:58.864115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:58.864082    5104 generic.go:356] "Generic (PLEG): container finished" podID="df0e75a3-027a-41f2-9e15-905c617efcd1" containerID="4be6d78ded1528d17a946da07d374205e4d6674d1009329b5133295c0c34544f" exitCode=0
Oct 30 01:00:58.864630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:00:58.864131    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw" event={"ID":"df0e75a3-027a-41f2-9e15-905c617efcd1","Type":"ContainerDied","Data":"4be6d78ded1528d17a946da07d374205e4d6674d1009329b5133295c0c34544f"}
Oct 30 01:01:00.068039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.068006    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:01:00.138215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.138121    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"]
Oct 30 01:01:00.139242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.139216    5104 cpu_manager.go:414] "containerMap: removing container" podUID="cd359f6a-f6c5-48f2-a822-5221332bb572" containerName="revoke-keys"
Oct 30 01:01:00.139242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.139243    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd359f6a-f6c5-48f2-a822-5221332bb572" containerName="revoke-keys"
Oct 30 01:01:00.139417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.139274    5104 cpu_manager.go:414] "containerMap: removing container" podUID="df0e75a3-027a-41f2-9e15-905c617efcd1" containerName="revoke-keys"
Oct 30 01:01:00.139417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.139281    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0e75a3-027a-41f2-9e15-905c617efcd1" containerName="revoke-keys"
Oct 30 01:01:00.139521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.139496    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd359f6a-f6c5-48f2-a822-5221332bb572" containerName="revoke-keys"
Oct 30 01:01:00.139521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.139517    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0e75a3-027a-41f2-9e15-905c617efcd1" containerName="revoke-keys"
Oct 30 01:01:00.144041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.144017    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.146869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.146840    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0e75a3-027a-41f2-9e15-905c617efcd1-kube-api-access-ctscg\" (UniqueName: \"kubernetes.io/projected/df0e75a3-027a-41f2-9e15-905c617efcd1-kube-api-access-ctscg\") pod \"df0e75a3-027a-41f2-9e15-905c617efcd1\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") "
Oct 30 01:01:00.147138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.147118    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-xks-ca\") pod \"df0e75a3-027a-41f2-9e15-905c617efcd1\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") "
Oct 30 01:01:00.147217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.147166    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-openshift-ca\") pod \"df0e75a3-027a-41f2-9e15-905c617efcd1\" (UID: \"df0e75a3-027a-41f2-9e15-905c617efcd1\") "
Oct 30 01:01:00.147628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.147602    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-openshift-ca" pod "df0e75a3-027a-41f2-9e15-905c617efcd1" (UID: "df0e75a3-027a-41f2-9e15-905c617efcd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:00.148068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.148049    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:00.149250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.149218    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-xks-ca" pod "df0e75a3-027a-41f2-9e15-905c617efcd1" (UID: "df0e75a3-027a-41f2-9e15-905c617efcd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:00.150348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.150310    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0e75a3-027a-41f2-9e15-905c617efcd1-kube-api-access-ctscg" pod "df0e75a3-027a-41f2-9e15-905c617efcd1" (UID: "df0e75a3-027a-41f2-9e15-905c617efcd1"). InnerVolumeSpecName "kube-api-access-ctscg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:00.156962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.156908    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"]
Oct 30 01:01:00.249857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.249820    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.250014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.249866    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.250014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.249994    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zbp\" (UniqueName: \"kubernetes.io/projected/6e707d38-566a-4080-ad70-7c186ced7bd5-kube-api-access-d6zbp\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.250122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.250110    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctscg\" (UniqueName: \"kubernetes.io/projected/df0e75a3-027a-41f2-9e15-905c617efcd1-kube-api-access-ctscg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:00.250156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.250125    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0e75a3-027a-41f2-9e15-905c617efcd1-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:00.351904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.351860    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zbp\" (UniqueName: \"kubernetes.io/projected/6e707d38-566a-4080-ad70-7c186ced7bd5-kube-api-access-d6zbp\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.352116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.352101    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.352175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.352127    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.352903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.352875    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.354563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.354540    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.369872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.369848    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zbp\" (UniqueName: \"kubernetes.io/projected/6e707d38-566a-4080-ad70-7c186ced7bd5-kube-api-access-d6zbp\") pod \"maas-api-revoke-sub-2408e466-7mkx8\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.508107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.508067    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:00.808577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.808544    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"]
Oct 30 01:01:00.810909 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:00.810809    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e707d38_566a_4080_ad70_7c186ced7bd5.slice/crio-8a00d0045b3d9c9673958f6a8c8f56937d405108d5b89d20242f40ae9eab09a2 WatchSource:0}: Error finding container 8a00d0045b3d9c9673958f6a8c8f56937d405108d5b89d20242f40ae9eab09a2: Status 404 returned error can't find the container with id 8a00d0045b3d9c9673958f6a8c8f56937d405108d5b89d20242f40ae9eab09a2
Oct 30 01:01:00.883065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.883027    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw" event={"ID":"df0e75a3-027a-41f2-9e15-905c617efcd1","Type":"ContainerDied","Data":"d50352f1d45e21d835fc168b2e196b15cbce2fd7604245baf34d1fe2c7009409"}
Oct 30 01:01:00.883260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.883092    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d50352f1d45e21d835fc168b2e196b15cbce2fd7604245baf34d1fe2c7009409"
Oct 30 01:01:00.883260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.883175    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"
Oct 30 01:01:00.885036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:00.885010    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8" event={"ID":"6e707d38-566a-4080-ad70-7c186ced7bd5","Type":"ContainerStarted","Data":"8a00d0045b3d9c9673958f6a8c8f56937d405108d5b89d20242f40ae9eab09a2"}
Oct 30 01:01:01.896533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:01.896499    5104 generic.go:356] "Generic (PLEG): container finished" podID="6e707d38-566a-4080-ad70-7c186ced7bd5" containerID="2bde5c094a688580a5450e4e6e247447177fee63488e56e288e11e7b489d3105" exitCode=0
Oct 30 01:01:01.896916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:01.896581    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8" event={"ID":"6e707d38-566a-4080-ad70-7c186ced7bd5","Type":"ContainerDied","Data":"2bde5c094a688580a5450e4e6e247447177fee63488e56e288e11e7b489d3105"}
Oct 30 01:01:01.949111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:01.949067    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"]
Oct 30 01:01:01.953691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:01.953665    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:01.969279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:01.969243    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"]
Oct 30 01:01:02.081400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.081298    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjq6c\" (UniqueName: \"kubernetes.io/projected/a155edeb-19fa-4020-ac91-c7a0a17bab59-kube-api-access-sjq6c\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.081586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.081565    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.081642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.081627    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.183427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.183381    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.183697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.183602    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.183775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.183733    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjq6c\" (UniqueName: \"kubernetes.io/projected/a155edeb-19fa-4020-ac91-c7a0a17bab59-kube-api-access-sjq6c\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.184334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.184308    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.186110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.186077    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.202124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.202093    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjq6c\" (UniqueName: \"kubernetes.io/projected/a155edeb-19fa-4020-ac91-c7a0a17bab59-kube-api-access-sjq6c\") pod \"maas-api-revoke-sub-4ba301f8-dj6q8\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.270908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.270873    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:02.405028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.404984    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:01:02.588171 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:02.588137    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda155edeb_19fa_4020_ac91_c7a0a17bab59.slice/crio-5466a887a6049001be8042fba2c476e041ee6287c18664537b24000977dbdd75 WatchSource:0}: Error finding container 5466a887a6049001be8042fba2c476e041ee6287c18664537b24000977dbdd75: Status 404 returned error can't find the container with id 5466a887a6049001be8042fba2c476e041ee6287c18664537b24000977dbdd75
Oct 30 01:01:02.590731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.590703    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"]
Oct 30 01:01:02.908338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:02.908258    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8" event={"ID":"a155edeb-19fa-4020-ac91-c7a0a17bab59","Type":"ContainerStarted","Data":"5466a887a6049001be8042fba2c476e041ee6287c18664537b24000977dbdd75"}
Oct 30 01:01:03.180544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.180513    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:03.322479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.320928    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-xks-ca\") pod \"6e707d38-566a-4080-ad70-7c186ced7bd5\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") "
Oct 30 01:01:03.322479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.321188    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-openshift-ca\") pod \"6e707d38-566a-4080-ad70-7c186ced7bd5\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") "
Oct 30 01:01:03.322479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.321271    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e707d38-566a-4080-ad70-7c186ced7bd5-kube-api-access-d6zbp\" (UniqueName: \"kubernetes.io/projected/6e707d38-566a-4080-ad70-7c186ced7bd5-kube-api-access-d6zbp\") pod \"6e707d38-566a-4080-ad70-7c186ced7bd5\" (UID: \"6e707d38-566a-4080-ad70-7c186ced7bd5\") "
Oct 30 01:01:03.336481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.335914    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-openshift-ca" pod "6e707d38-566a-4080-ad70-7c186ced7bd5" (UID: "6e707d38-566a-4080-ad70-7c186ced7bd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:03.365951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.362994    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-xks-ca" pod "6e707d38-566a-4080-ad70-7c186ced7bd5" (UID: "6e707d38-566a-4080-ad70-7c186ced7bd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:03.440671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.440580    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:03.440671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.440622    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e707d38-566a-4080-ad70-7c186ced7bd5-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:03.441433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.441379    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e707d38-566a-4080-ad70-7c186ced7bd5-kube-api-access-d6zbp" pod "6e707d38-566a-4080-ad70-7c186ced7bd5" (UID: "6e707d38-566a-4080-ad70-7c186ced7bd5"). InnerVolumeSpecName "kube-api-access-d6zbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:03.564482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.562990    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zbp\" (UniqueName: \"kubernetes.io/projected/6e707d38-566a-4080-ad70-7c186ced7bd5-kube-api-access-d6zbp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:03.941087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.941047    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8" event={"ID":"6e707d38-566a-4080-ad70-7c186ced7bd5","Type":"ContainerDied","Data":"8a00d0045b3d9c9673958f6a8c8f56937d405108d5b89d20242f40ae9eab09a2"}
Oct 30 01:01:03.941694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.941672    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a00d0045b3d9c9673958f6a8c8f56937d405108d5b89d20242f40ae9eab09a2"
Oct 30 01:01:03.941863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.941851    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"
Oct 30 01:01:03.944392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.944366    5104 generic.go:356] "Generic (PLEG): container finished" podID="a155edeb-19fa-4020-ac91-c7a0a17bab59" containerID="bf147eea7ff16949b155df573477ee7dd0a6b543542700f0738525a8cda144f6" exitCode=0
Oct 30 01:01:03.944743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:03.944721    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8" event={"ID":"a155edeb-19fa-4020-ac91-c7a0a17bab59","Type":"ContainerDied","Data":"bf147eea7ff16949b155df573477ee7dd0a6b543542700f0738525a8cda144f6"}
Oct 30 01:01:04.443803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:04.442551    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:04.827429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:04.827324    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:04.992550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:04.992500    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.065618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.065558    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.446048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.445999    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.446960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.446933    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:01:05.595621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.595555    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 30 01:01:05.772602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.772494    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:01:05.794726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.793712    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.810288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.810243    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.810881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.810849    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 01:01:05.841537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.827575    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.841537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.827986    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.854492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:05.842587    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:01:06.018936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.018881    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:06.106538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.073614    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:06.107900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.107854    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:06.123096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.109048    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:01:06.201516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.178039    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:01:06.288672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.242385    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:01:06.489908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.489858    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:01:06.831001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.830871    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:06.868173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.868136    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:01:06.889054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.889006    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:01:07.017484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.913172    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:07.017484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:06.969907    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:01:07.017484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.002607    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:07.147639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.111424    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:07.240482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.220576    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:07.376991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.376943    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:01:07.508499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.315567    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:07.508499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.497897    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:07.519093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.519051    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:01:07.699294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.634377    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:07.699294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.634447    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:07.699294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.635182    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:07.853540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.849525    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:07.992662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:07.992608    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:01:08.026330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.005598    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:01:08.050340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.005643    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:08.118233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.117391    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:08.398590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.389575    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:08.427196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.427160    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:08.482628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.482579    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:08.524230 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:01:08.524187    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.339s"
Oct 30 01:01:08.524491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.524445    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"]
Oct 30 01:01:08.542879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.542834    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:01:08.551547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.530062    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-xks-ca\") pod \"a155edeb-19fa-4020-ac91-c7a0a17bab59\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") "
Oct 30 01:01:08.551547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.543312    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a155edeb-19fa-4020-ac91-c7a0a17bab59-kube-api-access-sjq6c\" (UniqueName: \"kubernetes.io/projected/a155edeb-19fa-4020-ac91-c7a0a17bab59-kube-api-access-sjq6c\") pod \"a155edeb-19fa-4020-ac91-c7a0a17bab59\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") "
Oct 30 01:01:08.551547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.543348    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-openshift-ca\") pod \"a155edeb-19fa-4020-ac91-c7a0a17bab59\" (UID: \"a155edeb-19fa-4020-ac91-c7a0a17bab59\") "
Oct 30 01:01:08.552122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.552052    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-openshift-ca" pod "a155edeb-19fa-4020-ac91-c7a0a17bab59" (UID: "a155edeb-19fa-4020-ac91-c7a0a17bab59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:08.593494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.579945    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a155edeb-19fa-4020-ac91-c7a0a17bab59" containerName="revoke-keys"
Oct 30 01:01:08.594609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.594574    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a155edeb-19fa-4020-ac91-c7a0a17bab59" containerName="revoke-keys"
Oct 30 01:01:08.594831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.594816    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6e707d38-566a-4080-ad70-7c186ced7bd5" containerName="revoke-keys"
Oct 30 01:01:08.594920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.594908    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e707d38-566a-4080-ad70-7c186ced7bd5" containerName="revoke-keys"
Oct 30 01:01:08.595333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.595313    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a155edeb-19fa-4020-ac91-c7a0a17bab59" containerName="revoke-keys"
Oct 30 01:01:08.595478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.595447    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e707d38-566a-4080-ad70-7c186ced7bd5" containerName="revoke-keys"
Oct 30 01:01:08.623201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.623158    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-xks-ca" pod "a155edeb-19fa-4020-ac91-c7a0a17bab59" (UID: "a155edeb-19fa-4020-ac91-c7a0a17bab59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:08.757483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.676194    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.757483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.676260    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a155edeb-19fa-4020-ac91-c7a0a17bab59-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.757885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.757845    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:01:08.758501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.758449    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Oct 30 01:01:08.794489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.786613    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a155edeb-19fa-4020-ac91-c7a0a17bab59-kube-api-access-sjq6c" pod "a155edeb-19fa-4020-ac91-c7a0a17bab59" (UID: "a155edeb-19fa-4020-ac91-c7a0a17bab59"). InnerVolumeSpecName "kube-api-access-sjq6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.826811    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.826961    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827018    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d935bfd-813c-41f8-9721-b052f383076f-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827084    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9r7x\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-kube-api-access-w9r7x\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827154    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827265    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d935bfd-813c-41f8-9721-b052f383076f-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827314    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827382    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827490    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.827589    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjq6c\" (UniqueName: \"kubernetes.io/projected/a155edeb-19fa-4020-ac91-c7a0a17bab59-kube-api-access-sjq6c\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.878514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.847236    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"]
Oct 30 01:01:08.883485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.879364    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:01:08.941382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.894621    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:01:08.945918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.896027    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:08.949614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.949582    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:01:08.949823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.906391    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:08.953487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.949978    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:01:08.953740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:08.953720    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:01:09.036594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.036550    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:09.197914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.197874    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:09.199526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.199496    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:01:09.203930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.203890    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:09.338821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.281522    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.338821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338148    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d935bfd-813c-41f8-9721-b052f383076f-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.338821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338250    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9r7x\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-kube-api-access-w9r7x\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.338821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338374    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.338821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338586    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d935bfd-813c-41f8-9721-b052f383076f-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.338821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338660    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.338821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338740    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.339262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338874    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.339262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.338923    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.347490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.282152    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.376482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.372484    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"
Oct 30 01:01:09.421236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.379089    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.421801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.380033    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d935bfd-813c-41f8-9721-b052f383076f-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.421921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.380534    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.422004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.381042    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.442776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.404998    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.442993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.405046    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:09.492535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.489958    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:09.568707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.559600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:09.776582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.765448    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d935bfd-813c-41f8-9721-b052f383076f-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.776582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.771096    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.802485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.777471    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9r7x\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-kube-api-access-w9r7x\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:09.811555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.803322    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8" event={"ID":"a155edeb-19fa-4020-ac91-c7a0a17bab59","Type":"ContainerDied","Data":"5466a887a6049001be8042fba2c476e041ee6287c18664537b24000977dbdd75"}
Oct 30 01:01:09.811555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.803399    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5466a887a6049001be8042fba2c476e041ee6287c18664537b24000977dbdd75"
Oct 30 01:01:09.881242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.851950    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:09.941252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:09.941193    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.020482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.015264    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:10.102507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.047111    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.241980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.241930    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.259011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.258954    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.388337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.388291    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.445587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.445539    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.534638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.533971    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Oct 30 01:01:10.534638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.534290    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:10.622315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.620788    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.886152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:10.885984    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:10.909399 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:01:10.908360    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:01:11.160082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.159976    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:01:11.281695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.271973    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:11.418531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.418328    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:01:11.465535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.465480    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:11.535995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.535895    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:11.550221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.550185    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:01:11.563930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.563891    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:11.565553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.565518    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:01:11.565713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.565569    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"]
Oct 30 01:01:11.606790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.606752    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"]
Oct 30 01:01:11.606978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.606810    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"]
Oct 30 01:01:11.606978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.606808    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.613353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.613317    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"]
Oct 30 01:01:11.613514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.613396    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:01:11.613514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.613444    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:01:11.613514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.613496    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:01:11.613666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.613532    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:01:11.613666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.613567    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:01:11.614021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.614001    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:01:11.614103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.614056    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:01:11.614103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.614091    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:01:11.616620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.616598    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.621636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.621614    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:11.621760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.621745    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:01:11.621830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.621798    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:11.621899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.621886    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:11.655972 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:11.655940    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d935bfd_813c_41f8_9721_b052f383076f.slice/crio-171772511ce1a42f3d83850410d7393abb0a0ce9fa8aa7a0c8339ce3fcf4c79b WatchSource:0}: Error finding container 171772511ce1a42f3d83850410d7393abb0a0ce9fa8aa7a0c8339ce3fcf4c79b: Status 404 returned error can't find the container with id 171772511ce1a42f3d83850410d7393abb0a0ce9fa8aa7a0c8339ce3fcf4c79b
Oct 30 01:01:11.656186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.656170    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"]
Oct 30 01:01:11.658542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.658512    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:11.658647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.658574    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:11.658647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.658604    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:11.688367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.688336    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74lk6\" (UniqueName: \"kubernetes.io/projected/22657d84-b586-4fc8-8cb5-b9ab23415556-kube-api-access-74lk6\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.688733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.688707    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.688842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.688832    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.688968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.688948    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnp92\" (UniqueName: \"kubernetes.io/projected/db2d759f-e830-4661-bb7b-be778a34f031-kube-api-access-nnp92\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.689192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.689168    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.692051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.692028    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.795956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.795920    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lk6\" (UniqueName: \"kubernetes.io/projected/22657d84-b586-4fc8-8cb5-b9ab23415556-kube-api-access-74lk6\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.796172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.796058    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.796172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.796134    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.796274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.796208    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnp92\" (UniqueName: \"kubernetes.io/projected/db2d759f-e830-4661-bb7b-be778a34f031-kube-api-access-nnp92\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.796323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.796288    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.796404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.796386    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.799129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.799068    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.800656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.799523    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.812305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.812240    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.817833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.817773    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74lk6\" (UniqueName: \"kubernetes.io/projected/22657d84-b586-4fc8-8cb5-b9ab23415556-kube-api-access-74lk6\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.822194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.822141    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnp92\" (UniqueName: \"kubernetes.io/projected/db2d759f-e830-4661-bb7b-be778a34f031-kube-api-access-nnp92\") pod \"maas-api-revoke-sub-e03035d5-2bhns\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:11.830536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.830508    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04c40345-j4578\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.928898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.928868    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:11.936340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:11.936315    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:12.241632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.241535    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" event={"ID":"5d935bfd-813c-41f8-9721-b052f383076f","Type":"ContainerStarted","Data":"1d133be20ad9c919c5fa34de9e4e31708ce248f818cdac60caf97e1c688592b0"}
Oct 30 01:01:12.241632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.241605    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" event={"ID":"5d935bfd-813c-41f8-9721-b052f383076f","Type":"ContainerStarted","Data":"171772511ce1a42f3d83850410d7393abb0a0ce9fa8aa7a0c8339ce3fcf4c79b"}
Oct 30 01:01:12.287321 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:12.287292    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22657d84_b586_4fc8_8cb5_b9ab23415556.slice/crio-5147a838e74d2ad80fa0abdbe0fa0353f8e40e971cf88312fac0f38647b9de65 WatchSource:0}: Error finding container 5147a838e74d2ad80fa0abdbe0fa0353f8e40e971cf88312fac0f38647b9de65: Status 404 returned error can't find the container with id 5147a838e74d2ad80fa0abdbe0fa0353f8e40e971cf88312fac0f38647b9de65
Oct 30 01:01:12.287746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.287714    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"]
Oct 30 01:01:12.290649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.290522    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podStartSLOduration=6.290506793 podStartE2EDuration="6.290506793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:12.282722789 +0000 UTC m=+1113.830413290" watchObservedRunningTime="2026-10-30 01:01:12.290506793 +0000 UTC m=+1113.838197291"
Oct 30 01:01:12.301328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.301300    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"]
Oct 30 01:01:12.305679 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:12.305648    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2d759f_e830_4661_bb7b_be778a34f031.slice/crio-80600e58a99cb070cb33b6e787f5c14237f3e4d58181d0c6240d78fbf65d2a1e WatchSource:0}: Error finding container 80600e58a99cb070cb33b6e787f5c14237f3e4d58181d0c6240d78fbf65d2a1e: Status 404 returned error can't find the container with id 80600e58a99cb070cb33b6e787f5c14237f3e4d58181d0c6240d78fbf65d2a1e
Oct 30 01:01:12.505162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.505073    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"]
Oct 30 01:01:12.512602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.512570    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.519553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.519510    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"]
Oct 30 01:01:12.525740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.525367    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.529328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.529302    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"]
Oct 30 01:01:12.632314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.632251    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.632918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.632875    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2w95\" (UniqueName: \"kubernetes.io/projected/a9c5470c-11cf-4b28-850f-d985b4b84cbb-kube-api-access-l2w95\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.633038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.632946    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.633120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.633073    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwnn\" (UniqueName: \"kubernetes.io/projected/b5039c56-329d-478e-926a-c9cf551e043f-kube-api-access-qgwnn\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.633170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.633121    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.633170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.633154    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.653267 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.653234    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"]
Oct 30 01:01:12.739732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.739679    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.740413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.740381    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2w95\" (UniqueName: \"kubernetes.io/projected/a9c5470c-11cf-4b28-850f-d985b4b84cbb-kube-api-access-l2w95\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.740703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.740683    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.741087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.741046    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnn\" (UniqueName: \"kubernetes.io/projected/b5039c56-329d-478e-926a-c9cf551e043f-kube-api-access-qgwnn\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.741248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.741231    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.741475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.741368    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.745339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.745313    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.760921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.760839    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.773945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.773907    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.798479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.788772    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.801181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.801142    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2w95\" (UniqueName: \"kubernetes.io/projected/a9c5470c-11cf-4b28-850f-d985b4b84cbb-kube-api-access-l2w95\") pod \"maas-api-revoke-sub-d84afeab-k76fh\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:12.807512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.807470    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwnn\" (UniqueName: \"kubernetes.io/projected/b5039c56-329d-478e-926a-c9cf551e043f-kube-api-access-qgwnn\") pod \"maas-api-revoke-sub-396f4826-7wc58\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.837388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.837350    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:12.911845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:12.911805    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:13.032413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.032322    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:13.034109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.034067    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 01:01:13.034263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.034134    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 01:01:13.253446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.253403    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578" event={"ID":"22657d84-b586-4fc8-8cb5-b9ab23415556","Type":"ContainerStarted","Data":"706bffa71cca3d9d7240550567353b9637e03f762f3ece069caf95bcec417a9e"}
Oct 30 01:01:13.253744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.253714    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578" event={"ID":"22657d84-b586-4fc8-8cb5-b9ab23415556","Type":"ContainerStarted","Data":"5147a838e74d2ad80fa0abdbe0fa0353f8e40e971cf88312fac0f38647b9de65"}
Oct 30 01:01:13.255887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.255862    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns" event={"ID":"db2d759f-e830-4661-bb7b-be778a34f031","Type":"ContainerStarted","Data":"a3fd3c2bdad43e1caf870ae4fabec32d89dc9e309f0319b1249147ce4eb98895"}
Oct 30 01:01:13.256005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.255899    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns" event={"ID":"db2d759f-e830-4661-bb7b-be778a34f031","Type":"ContainerStarted","Data":"80600e58a99cb070cb33b6e787f5c14237f3e4d58181d0c6240d78fbf65d2a1e"}
Oct 30 01:01:13.329688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.322575    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578" podStartSLOduration=3.322552176 podStartE2EDuration="3.322552176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:13.289030063 +0000 UTC m=+1114.836720561" watchObservedRunningTime="2026-10-30 01:01:13.322552176 +0000 UTC m=+1114.870242674"
Oct 30 01:01:13.336239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.336219    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"]
Oct 30 01:01:13.341591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.341570    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"]
Oct 30 01:01:13.344100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:13.344049    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns" podStartSLOduration=3.344033869 podStartE2EDuration="3.344033869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:13.340618618 +0000 UTC m=+1114.888309126" watchObservedRunningTime="2026-10-30 01:01:13.344033869 +0000 UTC m=+1114.891724493"
Oct 30 01:01:14.032629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.032597    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 01:01:14.033065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.032651    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 01:01:14.266993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.266901    5104 generic.go:356] "Generic (PLEG): container finished" podID="22657d84-b586-4fc8-8cb5-b9ab23415556" containerID="706bffa71cca3d9d7240550567353b9637e03f762f3ece069caf95bcec417a9e" exitCode=0
Oct 30 01:01:14.267155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.267018    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578" event={"ID":"22657d84-b586-4fc8-8cb5-b9ab23415556","Type":"ContainerDied","Data":"706bffa71cca3d9d7240550567353b9637e03f762f3ece069caf95bcec417a9e"}
Oct 30 01:01:14.268627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.268588    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh" event={"ID":"a9c5470c-11cf-4b28-850f-d985b4b84cbb","Type":"ContainerStarted","Data":"a9b5b973f695f7a2bdf137674fa3a4e1fc582b27769a36f904d36ca480f8d45b"}
Oct 30 01:01:14.268771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.268646    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh" event={"ID":"a9c5470c-11cf-4b28-850f-d985b4b84cbb","Type":"ContainerStarted","Data":"dfead78337db6e4bdeefa43e964ffeca7a3379f0da7bc4ef4d2550619cb44cf1"}
Oct 30 01:01:14.270233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.270199    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58" event={"ID":"b5039c56-329d-478e-926a-c9cf551e043f","Type":"ContainerStarted","Data":"1d08a4f9f4cc17b7317a9b5bace93e60d7051a8338ec3d5d3c6cc3f8629c1830"}
Oct 30 01:01:14.270364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.270250    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58" event={"ID":"b5039c56-329d-478e-926a-c9cf551e043f","Type":"ContainerStarted","Data":"458466d293789d60f9ac0231ddfab1ea8f0fd4bc62511f5f08130af9063b77a4"}
Oct 30 01:01:14.271756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.271734    5104 generic.go:356] "Generic (PLEG): container finished" podID="db2d759f-e830-4661-bb7b-be778a34f031" containerID="a3fd3c2bdad43e1caf870ae4fabec32d89dc9e309f0319b1249147ce4eb98895" exitCode=0
Oct 30 01:01:14.271850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.271825    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns" event={"ID":"db2d759f-e830-4661-bb7b-be778a34f031","Type":"ContainerDied","Data":"a3fd3c2bdad43e1caf870ae4fabec32d89dc9e309f0319b1249147ce4eb98895"}
Oct 30 01:01:14.350132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.350077    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh" podStartSLOduration=2.350062276 podStartE2EDuration="2.350062276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:14.345888721 +0000 UTC m=+1115.893579217" watchObservedRunningTime="2026-10-30 01:01:14.350062276 +0000 UTC m=+1115.897752771"
Oct 30 01:01:14.419049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:14.418980    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58" podStartSLOduration=2.4189571770000002 podStartE2EDuration="2.418957177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:14.385792533 +0000 UTC m=+1115.933483056" watchObservedRunningTime="2026-10-30 01:01:14.418957177 +0000 UTC m=+1115.966647675"
Oct 30 01:01:15.032860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.032825    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 01:01:15.033355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.032890    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 01:01:15.325266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.325163    5104 generic.go:356] "Generic (PLEG): container finished" podID="a9c5470c-11cf-4b28-850f-d985b4b84cbb" containerID="a9b5b973f695f7a2bdf137674fa3a4e1fc582b27769a36f904d36ca480f8d45b" exitCode=0
Oct 30 01:01:15.325438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.325271    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh" event={"ID":"a9c5470c-11cf-4b28-850f-d985b4b84cbb","Type":"ContainerDied","Data":"a9b5b973f695f7a2bdf137674fa3a4e1fc582b27769a36f904d36ca480f8d45b"}
Oct 30 01:01:15.327563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.327538    5104 generic.go:356] "Generic (PLEG): container finished" podID="b5039c56-329d-478e-926a-c9cf551e043f" containerID="1d08a4f9f4cc17b7317a9b5bace93e60d7051a8338ec3d5d3c6cc3f8629c1830" exitCode=0
Oct 30 01:01:15.327689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.327620    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58" event={"ID":"b5039c56-329d-478e-926a-c9cf551e043f","Type":"ContainerDied","Data":"1d08a4f9f4cc17b7317a9b5bace93e60d7051a8338ec3d5d3c6cc3f8629c1830"}
Oct 30 01:01:15.425260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.425220    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"]
Oct 30 01:01:15.431779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.431756    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.444866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.444708    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"]
Oct 30 01:01:15.515261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.515198    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.515688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.515652    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnphc\" (UniqueName: \"kubernetes.io/projected/6c4db46f-0cdf-4caf-a510-3db194a43a14-kube-api-access-dnphc\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.516003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.515958    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.617517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.617481    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.617646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.617630    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnphc\" (UniqueName: \"kubernetes.io/projected/6c4db46f-0cdf-4caf-a510-3db194a43a14-kube-api-access-dnphc\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.617766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.617747    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.619124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.619101    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.620833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.620810    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.638270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.638241    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnphc\" (UniqueName: \"kubernetes.io/projected/6c4db46f-0cdf-4caf-a510-3db194a43a14-kube-api-access-dnphc\") pod \"maas-api-revoke-sub-acf40a09-c6btq\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.707258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.707230    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:15.710293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.710274    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:15.755610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.755572    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.829002    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-xks-ca\") pod \"db2d759f-e830-4661-bb7b-be778a34f031\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") "
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.829490    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-openshift-ca\") pod \"22657d84-b586-4fc8-8cb5-b9ab23415556\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") "
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.829614    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2d759f-e830-4661-bb7b-be778a34f031-kube-api-access-nnp92\" (UniqueName: \"kubernetes.io/projected/db2d759f-e830-4661-bb7b-be778a34f031-kube-api-access-nnp92\") pod \"db2d759f-e830-4661-bb7b-be778a34f031\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") "
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.829854    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-openshift-ca" pod "22657d84-b586-4fc8-8cb5-b9ab23415556" (UID: "22657d84-b586-4fc8-8cb5-b9ab23415556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.829923    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-openshift-ca\") pod \"db2d759f-e830-4661-bb7b-be778a34f031\" (UID: \"db2d759f-e830-4661-bb7b-be778a34f031\") "
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.829995    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-xks-ca\") pod \"22657d84-b586-4fc8-8cb5-b9ab23415556\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") "
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.830159    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22657d84-b586-4fc8-8cb5-b9ab23415556-kube-api-access-74lk6\" (UniqueName: \"kubernetes.io/projected/22657d84-b586-4fc8-8cb5-b9ab23415556-kube-api-access-74lk6\") pod \"22657d84-b586-4fc8-8cb5-b9ab23415556\" (UID: \"22657d84-b586-4fc8-8cb5-b9ab23415556\") "
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.830926    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.831629    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-xks-ca" pod "db2d759f-e830-4661-bb7b-be778a34f031" (UID: "db2d759f-e830-4661-bb7b-be778a34f031"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:15.832157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.831694    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-openshift-ca" pod "db2d759f-e830-4661-bb7b-be778a34f031" (UID: "db2d759f-e830-4661-bb7b-be778a34f031"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:15.843662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.843602    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2d759f-e830-4661-bb7b-be778a34f031-kube-api-access-nnp92" pod "db2d759f-e830-4661-bb7b-be778a34f031" (UID: "db2d759f-e830-4661-bb7b-be778a34f031"). InnerVolumeSpecName "kube-api-access-nnp92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:15.843662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.843641    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22657d84-b586-4fc8-8cb5-b9ab23415556-kube-api-access-74lk6" pod "22657d84-b586-4fc8-8cb5-b9ab23415556" (UID: "22657d84-b586-4fc8-8cb5-b9ab23415556"). InnerVolumeSpecName "kube-api-access-74lk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:15.847608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.847569    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-xks-ca" pod "22657d84-b586-4fc8-8cb5-b9ab23415556" (UID: "22657d84-b586-4fc8-8cb5-b9ab23415556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:15.932143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.932106    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnp92\" (UniqueName: \"kubernetes.io/projected/db2d759f-e830-4661-bb7b-be778a34f031-kube-api-access-nnp92\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:15.932143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.932141    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:15.932329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.932155    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22657d84-b586-4fc8-8cb5-b9ab23415556-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:15.932329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.932169    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74lk6\" (UniqueName: \"kubernetes.io/projected/22657d84-b586-4fc8-8cb5-b9ab23415556-kube-api-access-74lk6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:15.932329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:15.932181    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2d759f-e830-4661-bb7b-be778a34f031-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:16.032294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.032255    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 01:01:16.032437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.032324    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 01:01:16.131768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.131737    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"]
Oct 30 01:01:16.132120 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:16.131871    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4db46f_0cdf_4caf_a510_3db194a43a14.slice/crio-1c41bb7c38eaadecf6816631e855522525dac8a81df6b6ad6698638629dd63c1 WatchSource:0}: Error finding container 1c41bb7c38eaadecf6816631e855522525dac8a81df6b6ad6698638629dd63c1: Status 404 returned error can't find the container with id 1c41bb7c38eaadecf6816631e855522525dac8a81df6b6ad6698638629dd63c1
Oct 30 01:01:16.412403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.412363    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq" event={"ID":"6c4db46f-0cdf-4caf-a510-3db194a43a14","Type":"ContainerStarted","Data":"1c41bb7c38eaadecf6816631e855522525dac8a81df6b6ad6698638629dd63c1"}
Oct 30 01:01:16.414843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.414803    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"
Oct 30 01:01:16.414843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.414831    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578" event={"ID":"22657d84-b586-4fc8-8cb5-b9ab23415556","Type":"ContainerDied","Data":"5147a838e74d2ad80fa0abdbe0fa0353f8e40e971cf88312fac0f38647b9de65"}
Oct 30 01:01:16.414997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.414875    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5147a838e74d2ad80fa0abdbe0fa0353f8e40e971cf88312fac0f38647b9de65"
Oct 30 01:01:16.417120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.417100    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"
Oct 30 01:01:16.417209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.417172    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns" event={"ID":"db2d759f-e830-4661-bb7b-be778a34f031","Type":"ContainerDied","Data":"80600e58a99cb070cb33b6e787f5c14237f3e4d58181d0c6240d78fbf65d2a1e"}
Oct 30 01:01:16.417276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.417215    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80600e58a99cb070cb33b6e787f5c14237f3e4d58181d0c6240d78fbf65d2a1e"
Oct 30 01:01:16.713767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.713737    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:16.769707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.769679    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:16.869054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.869002    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-xks-ca\") pod \"b5039c56-329d-478e-926a-c9cf551e043f\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") "
Oct 30 01:01:16.869263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.869202    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-openshift-ca\") pod \"b5039c56-329d-478e-926a-c9cf551e043f\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") "
Oct 30 01:01:16.869263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.869242    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-openshift-ca\") pod \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") "
Oct 30 01:01:16.869379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.869297    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-xks-ca\") pod \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") "
Oct 30 01:01:16.869428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.869388    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5039c56-329d-478e-926a-c9cf551e043f-kube-api-access-qgwnn\" (UniqueName: \"kubernetes.io/projected/b5039c56-329d-478e-926a-c9cf551e043f-kube-api-access-qgwnn\") pod \"b5039c56-329d-478e-926a-c9cf551e043f\" (UID: \"b5039c56-329d-478e-926a-c9cf551e043f\") "
Oct 30 01:01:16.869499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.869441    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c5470c-11cf-4b28-850f-d985b4b84cbb-kube-api-access-l2w95\" (UniqueName: \"kubernetes.io/projected/a9c5470c-11cf-4b28-850f-d985b4b84cbb-kube-api-access-l2w95\") pod \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\" (UID: \"a9c5470c-11cf-4b28-850f-d985b4b84cbb\") "
Oct 30 01:01:16.869747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.869714    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-openshift-ca" pod "b5039c56-329d-478e-926a-c9cf551e043f" (UID: "b5039c56-329d-478e-926a-c9cf551e043f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:16.870180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.870147    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:16.870766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.870739    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-openshift-ca" pod "a9c5470c-11cf-4b28-850f-d985b4b84cbb" (UID: "a9c5470c-11cf-4b28-850f-d985b4b84cbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:16.872574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.872535    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-xks-ca" pod "b5039c56-329d-478e-926a-c9cf551e043f" (UID: "b5039c56-329d-478e-926a-c9cf551e043f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:16.877499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.873013    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5039c56-329d-478e-926a-c9cf551e043f-kube-api-access-qgwnn" pod "b5039c56-329d-478e-926a-c9cf551e043f" (UID: "b5039c56-329d-478e-926a-c9cf551e043f"). InnerVolumeSpecName "kube-api-access-qgwnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:16.908480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.900589    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-xks-ca" pod "a9c5470c-11cf-4b28-850f-d985b4b84cbb" (UID: "a9c5470c-11cf-4b28-850f-d985b4b84cbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:16.910389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.908867    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c5470c-11cf-4b28-850f-d985b4b84cbb-kube-api-access-l2w95" pod "a9c5470c-11cf-4b28-850f-d985b4b84cbb" (UID: "a9c5470c-11cf-4b28-850f-d985b4b84cbb"). InnerVolumeSpecName "kube-api-access-l2w95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:16.972003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.971917    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:16.972003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.971951    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5470c-11cf-4b28-850f-d985b4b84cbb-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:16.972003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.971960    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwnn\" (UniqueName: \"kubernetes.io/projected/b5039c56-329d-478e-926a-c9cf551e043f-kube-api-access-qgwnn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:16.972003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.971969    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2w95\" (UniqueName: \"kubernetes.io/projected/a9c5470c-11cf-4b28-850f-d985b4b84cbb-kube-api-access-l2w95\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:16.972003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:16.971977    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5039c56-329d-478e-926a-c9cf551e043f-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:17.032785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.032752    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 01:01:17.032946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.032803    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 01:01:17.436162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.436120    5104 generic.go:356] "Generic (PLEG): container finished" podID="6c4db46f-0cdf-4caf-a510-3db194a43a14" containerID="a351bf1ee3e3104b3b819e715f2fc6fb615db6c4cf8e9f57f404efab7225d356" exitCode=0
Oct 30 01:01:17.436659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.436211    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq" event={"ID":"6c4db46f-0cdf-4caf-a510-3db194a43a14","Type":"ContainerDied","Data":"a351bf1ee3e3104b3b819e715f2fc6fb615db6c4cf8e9f57f404efab7225d356"}
Oct 30 01:01:17.438155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.438131    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"
Oct 30 01:01:17.438284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.438144    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh" event={"ID":"a9c5470c-11cf-4b28-850f-d985b4b84cbb","Type":"ContainerDied","Data":"dfead78337db6e4bdeefa43e964ffeca7a3379f0da7bc4ef4d2550619cb44cf1"}
Oct 30 01:01:17.438284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.438194    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfead78337db6e4bdeefa43e964ffeca7a3379f0da7bc4ef4d2550619cb44cf1"
Oct 30 01:01:17.440252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.440222    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58" event={"ID":"b5039c56-329d-478e-926a-c9cf551e043f","Type":"ContainerDied","Data":"458466d293789d60f9ac0231ddfab1ea8f0fd4bc62511f5f08130af9063b77a4"}
Oct 30 01:01:17.440396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.440273    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="458466d293789d60f9ac0231ddfab1ea8f0fd4bc62511f5f08130af9063b77a4"
Oct 30 01:01:17.440396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:17.440344    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"
Oct 30 01:01:18.041479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.040665    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 01:01:18.041479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.040724    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 01:01:18.652550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.652518    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:18.750678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.750568    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-xks-ca\") pod \"6c4db46f-0cdf-4caf-a510-3db194a43a14\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") "
Oct 30 01:01:18.750861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.750710    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-openshift-ca\") pod \"6c4db46f-0cdf-4caf-a510-3db194a43a14\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") "
Oct 30 01:01:18.751138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.751107    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-openshift-ca" pod "6c4db46f-0cdf-4caf-a510-3db194a43a14" (UID: "6c4db46f-0cdf-4caf-a510-3db194a43a14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:18.751278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.751257    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4db46f-0cdf-4caf-a510-3db194a43a14-kube-api-access-dnphc\" (UniqueName: \"kubernetes.io/projected/6c4db46f-0cdf-4caf-a510-3db194a43a14-kube-api-access-dnphc\") pod \"6c4db46f-0cdf-4caf-a510-3db194a43a14\" (UID: \"6c4db46f-0cdf-4caf-a510-3db194a43a14\") "
Oct 30 01:01:18.752405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.752375    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:18.755117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.755086    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4db46f-0cdf-4caf-a510-3db194a43a14-kube-api-access-dnphc" pod "6c4db46f-0cdf-4caf-a510-3db194a43a14" (UID: "6c4db46f-0cdf-4caf-a510-3db194a43a14"). InnerVolumeSpecName "kube-api-access-dnphc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:18.755230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.755202    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-xks-ca" pod "6c4db46f-0cdf-4caf-a510-3db194a43a14" (UID: "6c4db46f-0cdf-4caf-a510-3db194a43a14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:18.853714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.853673    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnphc\" (UniqueName: \"kubernetes.io/projected/6c4db46f-0cdf-4caf-a510-3db194a43a14-kube-api-access-dnphc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:18.853714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:18.853711    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4db46f-0cdf-4caf-a510-3db194a43a14-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:19.032067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:19.031974    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 01:01:19.032067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:19.032025    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 01:01:19.462521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:19.462449    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq" event={"ID":"6c4db46f-0cdf-4caf-a510-3db194a43a14","Type":"ContainerDied","Data":"1c41bb7c38eaadecf6816631e855522525dac8a81df6b6ad6698638629dd63c1"}
Oct 30 01:01:19.462740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:19.462534    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c41bb7c38eaadecf6816631e855522525dac8a81df6b6ad6698638629dd63c1"
Oct 30 01:01:19.462740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:19.462581    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"
Oct 30 01:01:20.037526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:20.037496    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:20.038027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:20.038002    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:20.038726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:20.038706    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:01:23.875836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.875796    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"]
Oct 30 01:01:23.876873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876849    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b5039c56-329d-478e-926a-c9cf551e043f" containerName="revoke-keys"
Oct 30 01:01:23.876873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876872    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5039c56-329d-478e-926a-c9cf551e043f" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876908    5104 cpu_manager.go:414] "containerMap: removing container" podUID="22657d84-b586-4fc8-8cb5-b9ab23415556" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876916    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22657d84-b586-4fc8-8cb5-b9ab23415556" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876956    5104 cpu_manager.go:414] "containerMap: removing container" podUID="db2d759f-e830-4661-bb7b-be778a34f031" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876964    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2d759f-e830-4661-bb7b-be778a34f031" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876986    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a9c5470c-11cf-4b28-850f-d985b4b84cbb" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.876992    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c5470c-11cf-4b28-850f-d985b4b84cbb" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.877023    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6c4db46f-0cdf-4caf-a510-3db194a43a14" containerName="revoke-keys"
Oct 30 01:01:23.877084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.877030    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4db46f-0cdf-4caf-a510-3db194a43a14" containerName="revoke-keys"
Oct 30 01:01:23.877474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.877187    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2d759f-e830-4661-bb7b-be778a34f031" containerName="revoke-keys"
Oct 30 01:01:23.877474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.877201    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c5470c-11cf-4b28-850f-d985b4b84cbb" containerName="revoke-keys"
Oct 30 01:01:23.877474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.877218    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4db46f-0cdf-4caf-a510-3db194a43a14" containerName="revoke-keys"
Oct 30 01:01:23.877474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.877232    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5039c56-329d-478e-926a-c9cf551e043f" containerName="revoke-keys"
Oct 30 01:01:23.877474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.877244    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="22657d84-b586-4fc8-8cb5-b9ab23415556" containerName="revoke-keys"
Oct 30 01:01:23.887980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.883334    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:23.899749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.899721    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"]
Oct 30 01:01:23.903297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.903267    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:23.903438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.903310    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:01:23.903438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.903425    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:23.920508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.916647    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:23.942774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.942739    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:23.943300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.943266    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:23.943430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:23.943342    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4nh\" (UniqueName: \"kubernetes.io/projected/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-kube-api-access-lt4nh\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.045011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.044969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.045185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.045104    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.045185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.045135    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4nh\" (UniqueName: \"kubernetes.io/projected/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-kube-api-access-lt4nh\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.045870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.045847    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.047404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.047385    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.062787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.062758    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4nh\" (UniqueName: \"kubernetes.io/projected/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-kube-api-access-lt4nh\") pod \"maas-api-revoke-sub-6bf3e35d-tk52c\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.202862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.202825    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:24.512225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:24.512193    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"]
Oct 30 01:01:24.513252 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:24.513218    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745dead4_1e2e_4a4f_9f67_d8a6326ffa59.slice/crio-c7491b0d240070694f78fd2f6bbec081fe98a3b07d01c3cea67be3659b9eca64 WatchSource:0}: Error finding container c7491b0d240070694f78fd2f6bbec081fe98a3b07d01c3cea67be3659b9eca64: Status 404 returned error can't find the container with id c7491b0d240070694f78fd2f6bbec081fe98a3b07d01c3cea67be3659b9eca64
Oct 30 01:01:25.526029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:25.525989    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c" event={"ID":"745dead4-1e2e-4a4f-9f67-d8a6326ffa59","Type":"ContainerStarted","Data":"08cc8b01e3145d361d6e341252cc20611074b7f8d2dfdcb02c1b28d1901c9295"}
Oct 30 01:01:25.526029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:25.526047    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c" event={"ID":"745dead4-1e2e-4a4f-9f67-d8a6326ffa59","Type":"ContainerStarted","Data":"c7491b0d240070694f78fd2f6bbec081fe98a3b07d01c3cea67be3659b9eca64"}
Oct 30 01:01:25.563993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:25.563946    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c" podStartSLOduration=2.563932293 podStartE2EDuration="2.563932293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:25.561944254 +0000 UTC m=+1127.109634750" watchObservedRunningTime="2026-10-30 01:01:25.563932293 +0000 UTC m=+1127.111622795"
Oct 30 01:01:26.543563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:26.543530    5104 generic.go:356] "Generic (PLEG): container finished" podID="745dead4-1e2e-4a4f-9f67-d8a6326ffa59" containerID="08cc8b01e3145d361d6e341252cc20611074b7f8d2dfdcb02c1b28d1901c9295" exitCode=0
Oct 30 01:01:26.543983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:26.543620    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c" event={"ID":"745dead4-1e2e-4a4f-9f67-d8a6326ffa59","Type":"ContainerDied","Data":"08cc8b01e3145d361d6e341252cc20611074b7f8d2dfdcb02c1b28d1901c9295"}
Oct 30 01:01:27.736092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.736059    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"]
Oct 30 01:01:27.741074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.741054    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.748853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.748835    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:27.758966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.758943    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"]
Oct 30 01:01:27.815805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.815698    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-openshift-ca\") pod \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") "
Oct 30 01:01:27.816252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.816230    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-kube-api-access-lt4nh\" (UniqueName: \"kubernetes.io/projected/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-kube-api-access-lt4nh\") pod \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") "
Oct 30 01:01:27.817616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.817154    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-xks-ca\") pod \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\" (UID: \"745dead4-1e2e-4a4f-9f67-d8a6326ffa59\") "
Oct 30 01:01:27.818216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.818190    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.818995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.818976    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.819443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.819423    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2jv\" (UniqueName: \"kubernetes.io/projected/925f65e7-de90-43a4-8903-e0e0e530d056-kube-api-access-lr2jv\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.825648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.825612    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-kube-api-access-lt4nh" pod "745dead4-1e2e-4a4f-9f67-d8a6326ffa59" (UID: "745dead4-1e2e-4a4f-9f67-d8a6326ffa59"). InnerVolumeSpecName "kube-api-access-lt4nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:27.827546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.827519    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-openshift-ca" pod "745dead4-1e2e-4a4f-9f67-d8a6326ffa59" (UID: "745dead4-1e2e-4a4f-9f67-d8a6326ffa59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:27.830393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.830364    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-xks-ca" pod "745dead4-1e2e-4a4f-9f67-d8a6326ffa59" (UID: "745dead4-1e2e-4a4f-9f67-d8a6326ffa59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:27.857443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.857407    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"]
Oct 30 01:01:27.858513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.858491    5104 cpu_manager.go:414] "containerMap: removing container" podUID="745dead4-1e2e-4a4f-9f67-d8a6326ffa59" containerName="revoke-keys"
Oct 30 01:01:27.858672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.858643    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745dead4-1e2e-4a4f-9f67-d8a6326ffa59" containerName="revoke-keys"
Oct 30 01:01:27.858933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.858916    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="745dead4-1e2e-4a4f-9f67-d8a6326ffa59" containerName="revoke-keys"
Oct 30 01:01:27.864046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.864031    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:27.893497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.893471    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"]
Oct 30 01:01:27.921174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.921138    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2jv\" (UniqueName: \"kubernetes.io/projected/925f65e7-de90-43a4-8903-e0e0e530d056-kube-api-access-lr2jv\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.921497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.921479    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.921769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.921751    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.922048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.922034    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.922691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.922670    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.923563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.923538    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt4nh\" (UniqueName: \"kubernetes.io/projected/745dead4-1e2e-4a4f-9f67-d8a6326ffa59-kube-api-access-lt4nh\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.925287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.925247    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.925393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.925319    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:27.941097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:27.941064    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2jv\" (UniqueName: \"kubernetes.io/projected/925f65e7-de90-43a4-8903-e0e0e530d056-kube-api-access-lr2jv\") pod \"maas-api-revoke-sub-abdd57be-tmfsl\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:28.024747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.024710    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.025001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.024977    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.025085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.025021    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgn4v\" (UniqueName: \"kubernetes.io/projected/cbf18d57-9466-4bc6-a645-08d241c4061f-kube-api-access-sgn4v\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.059451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.059421    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:28.126680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.126647    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.129413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.126920    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.129413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.126969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgn4v\" (UniqueName: \"kubernetes.io/projected/cbf18d57-9466-4bc6-a645-08d241c4061f-kube-api-access-sgn4v\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.129413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.127856    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.130357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.130031    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.145579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.145537    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgn4v\" (UniqueName: \"kubernetes.io/projected/cbf18d57-9466-4bc6-a645-08d241c4061f-kube-api-access-sgn4v\") pod \"maas-api-revoke-sub-99376cb4-25l7k\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.202258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.202227    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:28.366803 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:28.366750    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925f65e7_de90_43a4_8903_e0e0e530d056.slice/crio-7be1b78006dfbac4bfbe5d32b57c3ebcdeee325e0a97dd7f9d1c64b1bfa89710 WatchSource:0}: Error finding container 7be1b78006dfbac4bfbe5d32b57c3ebcdeee325e0a97dd7f9d1c64b1bfa89710: Status 404 returned error can't find the container with id 7be1b78006dfbac4bfbe5d32b57c3ebcdeee325e0a97dd7f9d1c64b1bfa89710
Oct 30 01:01:28.368468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.368427    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"]
Oct 30 01:01:28.614725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.596156    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"]
Oct 30 01:01:28.682977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.682939    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"
Oct 30 01:01:28.684512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.684137    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c" event={"ID":"745dead4-1e2e-4a4f-9f67-d8a6326ffa59","Type":"ContainerDied","Data":"c7491b0d240070694f78fd2f6bbec081fe98a3b07d01c3cea67be3659b9eca64"}
Oct 30 01:01:28.684512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.684196    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7491b0d240070694f78fd2f6bbec081fe98a3b07d01c3cea67be3659b9eca64"
Oct 30 01:01:28.685196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.685169    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl" event={"ID":"925f65e7-de90-43a4-8903-e0e0e530d056","Type":"ContainerStarted","Data":"7be1b78006dfbac4bfbe5d32b57c3ebcdeee325e0a97dd7f9d1c64b1bfa89710"}
Oct 30 01:01:28.930501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.930449    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"]
Oct 30 01:01:28.936220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.936191    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:28.951878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:28.951852    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"]
Oct 30 01:01:29.079920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.079883    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqs89\" (UniqueName: \"kubernetes.io/projected/09307ada-bbab-4ea2-8c61-86e50a8d2c78-kube-api-access-hqs89\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.097272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.079937    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.097272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.080064    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.181983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.181949    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqs89\" (UniqueName: \"kubernetes.io/projected/09307ada-bbab-4ea2-8c61-86e50a8d2c78-kube-api-access-hqs89\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.182204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.182001    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.182204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.182041    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.182759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.182732    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.184718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.184683    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.200053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.200024    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqs89\" (UniqueName: \"kubernetes.io/projected/09307ada-bbab-4ea2-8c61-86e50a8d2c78-kube-api-access-hqs89\") pod \"maas-api-revoke-sub-801fc8cf-rfckg\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.260992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.260956    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:29.566752 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:29.566708    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09307ada_bbab_4ea2_8c61_86e50a8d2c78.slice/crio-ad1444a123a2e5093337f45ccb758da9da81ef037f479b2783aae5f18ef8fe78 WatchSource:0}: Error finding container ad1444a123a2e5093337f45ccb758da9da81ef037f479b2783aae5f18ef8fe78: Status 404 returned error can't find the container with id ad1444a123a2e5093337f45ccb758da9da81ef037f479b2783aae5f18ef8fe78
Oct 30 01:01:29.568294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.568268    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"]
Oct 30 01:01:29.697925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.697887    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k" event={"ID":"cbf18d57-9466-4bc6-a645-08d241c4061f","Type":"ContainerStarted","Data":"16b847c7233baab699fefc2ba67e9ceb4c4e5a88710733b8e3c7a9915b8ae954"}
Oct 30 01:01:29.698104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.697957    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k" event={"ID":"cbf18d57-9466-4bc6-a645-08d241c4061f","Type":"ContainerStarted","Data":"aee73bc1ca65a2da758fdc07165ec455f7e9d1027b98c0779d2ca3774309fbc8"}
Oct 30 01:01:29.700351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.700323    5104 generic.go:356] "Generic (PLEG): container finished" podID="925f65e7-de90-43a4-8903-e0e0e530d056" containerID="33d9c6781bbdae9a8d5c786fc0d20074a4c88f61d3a8f7b77b41e5b381b23560" exitCode=0
Oct 30 01:01:29.700504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.700409    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl" event={"ID":"925f65e7-de90-43a4-8903-e0e0e530d056","Type":"ContainerDied","Data":"33d9c6781bbdae9a8d5c786fc0d20074a4c88f61d3a8f7b77b41e5b381b23560"}
Oct 30 01:01:29.701909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.701887    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg" event={"ID":"09307ada-bbab-4ea2-8c61-86e50a8d2c78","Type":"ContainerStarted","Data":"ad1444a123a2e5093337f45ccb758da9da81ef037f479b2783aae5f18ef8fe78"}
Oct 30 01:01:29.739239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:29.738123    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k" podStartSLOduration=2.738103497 podStartE2EDuration="2.738103497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:29.734444619 +0000 UTC m=+1131.282135136" watchObservedRunningTime="2026-10-30 01:01:29.738103497 +0000 UTC m=+1131.285793995"
Oct 30 01:01:30.880483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:30.875883    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg" event={"ID":"09307ada-bbab-4ea2-8c61-86e50a8d2c78","Type":"ContainerStarted","Data":"821d5ecd67198fa10c43985a980156098157562184d44478521539c3eacc5c5c"}
Oct 30 01:01:30.930617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:30.902219    5104 generic.go:356] "Generic (PLEG): container finished" podID="cbf18d57-9466-4bc6-a645-08d241c4061f" containerID="16b847c7233baab699fefc2ba67e9ceb4c4e5a88710733b8e3c7a9915b8ae954" exitCode=0
Oct 30 01:01:30.930617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:30.902774    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k" event={"ID":"cbf18d57-9466-4bc6-a645-08d241c4061f","Type":"ContainerDied","Data":"16b847c7233baab699fefc2ba67e9ceb4c4e5a88710733b8e3c7a9915b8ae954"}
Oct 30 01:01:31.036534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.033242    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg" podStartSLOduration=3.033218113 podStartE2EDuration="3.033218113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:30.916342131 +0000 UTC m=+1132.464032627" watchObservedRunningTime="2026-10-30 01:01:31.033218113 +0000 UTC m=+1132.580908609"
Oct 30 01:01:31.293542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.268600    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Oct 30 01:01:31.297480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.294819    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:31.417836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.417779    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:31.464575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.464520    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:31.493063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.493010    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:31.598825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.598723    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:31.599429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.599398    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:01:31.902497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:31.900603    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.227492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.220016    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:01:32.344523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.344472    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.345111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.344958    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:01:32.345593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.345569    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:01:32.403419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.403382    5104 generic.go:356] "Generic (PLEG): container finished" podID="09307ada-bbab-4ea2-8c61-86e50a8d2c78" containerID="821d5ecd67198fa10c43985a980156098157562184d44478521539c3eacc5c5c" exitCode=0
Oct 30 01:01:32.602523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.425352    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg" event={"ID":"09307ada-bbab-4ea2-8c61-86e50a8d2c78","Type":"ContainerDied","Data":"821d5ecd67198fa10c43985a980156098157562184d44478521539c3eacc5c5c"}
Oct 30 01:01:32.649842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.426085    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.661503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.487661    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.661503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.515573    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.661503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.630270    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.661503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.630312    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:01:32.688891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.688815    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.844246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.844160    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:32.906913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.850580    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-xks-ca\") pod \"925f65e7-de90-43a4-8903-e0e0e530d056\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") "
Oct 30 01:01:32.906913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.850722    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-openshift-ca\") pod \"925f65e7-de90-43a4-8903-e0e0e530d056\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") "
Oct 30 01:01:32.906913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.850773    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925f65e7-de90-43a4-8903-e0e0e530d056-kube-api-access-lr2jv\" (UniqueName: \"kubernetes.io/projected/925f65e7-de90-43a4-8903-e0e0e530d056-kube-api-access-lr2jv\") pod \"925f65e7-de90-43a4-8903-e0e0e530d056\" (UID: \"925f65e7-de90-43a4-8903-e0e0e530d056\") "
Oct 30 01:01:32.916482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.912647    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:32.916482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.912745    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:32.938857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.921912    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-openshift-ca" pod "925f65e7-de90-43a4-8903-e0e0e530d056" (UID: "925f65e7-de90-43a4-8903-e0e0e530d056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:32.938857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:32.922030    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:33.060496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.034266    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:33.373694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.315534    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:33.373694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.373557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:33.489607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.478914    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.628779    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.628832    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.628866    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.628902    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.628934    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.628965    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.628997    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.553686    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-xks-ca" pod "925f65e7-de90-43a4-8903-e0e0e530d056" (UID: "925f65e7-de90-43a4-8903-e0e0e530d056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:33.633538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.374260    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:33.644323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.644285    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925f65e7-de90-43a4-8903-e0e0e530d056-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:33.690379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.690309    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:33.703023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.702988    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"
Oct 30 01:01:33.741360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.730229    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:33.741360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.732664    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl" event={"ID":"925f65e7-de90-43a4-8903-e0e0e530d056","Type":"ContainerDied","Data":"7be1b78006dfbac4bfbe5d32b57c3ebcdeee325e0a97dd7f9d1c64b1bfa89710"}
Oct 30 01:01:33.741360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.732729    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7be1b78006dfbac4bfbe5d32b57c3ebcdeee325e0a97dd7f9d1c64b1bfa89710"
Oct 30 01:01:33.741360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.734482    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:33.756483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.751883    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:01:33.782482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.778651    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:01:33.833650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.833606    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925f65e7-de90-43a4-8903-e0e0e530d056-kube-api-access-lr2jv" pod "925f65e7-de90-43a4-8903-e0e0e530d056" (UID: "925f65e7-de90-43a4-8903-e0e0e530d056"). InnerVolumeSpecName "kube-api-access-lr2jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:33.876482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:33.875930    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2jv\" (UniqueName: \"kubernetes.io/projected/925f65e7-de90-43a4-8903-e0e0e530d056-kube-api-access-lr2jv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:34.030514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.018184    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:01:34.330575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.320538    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:34.467481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.465756    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:01:34.641817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.641710    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:34.698660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.698605    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:34.917500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.775816    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:34.947880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.775854    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:35.014482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.775915    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.018480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.793046    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:35.032482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.028638    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.105065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.028554    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:01:35.105746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:34.792990    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.116614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.109861    5104 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:35.116614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.109912    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.467066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.467015    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.560336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.560278    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:35.560637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.560601    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.587092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.587038    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.646483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.615924    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 30 01:01:35.727779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.727672    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:35.740116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.740054    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.765971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.765928    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:35.850073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.850027    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 01:01:35.984222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:35.984124    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:36.078562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.019072    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:36.078562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.019377    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:01:36.078562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.033361    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:36.189587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.189522    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:36.281570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.280846    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:36.431227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.282291    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 01:01:36.437188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.400072    5104 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:36.461483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.437582    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:36.561677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.534542    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:01:36.561677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.534767    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 01:01:36.561677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.535321    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:01:36.599635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.599566    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:36.866564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:36.855262    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:37.021486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.021425    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:37.050510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.047276    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:01:37.050510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.047336    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:01:37.050510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.047368    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:01:37.416572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.281731    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:37.450562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.449616    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"]
Oct 30 01:01:37.491155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.491069    5104 cpu_manager.go:414] "containerMap: removing container" podUID="925f65e7-de90-43a4-8903-e0e0e530d056" containerName="revoke-keys"
Oct 30 01:01:37.491353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.491339    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925f65e7-de90-43a4-8903-e0e0e530d056" containerName="revoke-keys"
Oct 30 01:01:37.491767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.491750    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="925f65e7-de90-43a4-8903-e0e0e530d056" containerName="revoke-keys"
Oct 30 01:01:37.500761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.500718    5104 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:01:37.585543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.558435    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:37.585543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.575423    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:37.615520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.602645    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:37.626512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.626440    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:01:37.639665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.639626    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:37.660923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.641741    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:37.673985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.673960    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:01:37.760096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.760001    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09307ada-bbab-4ea2-8c61-86e50a8d2c78-kube-api-access-hqs89\" (UniqueName: \"kubernetes.io/projected/09307ada-bbab-4ea2-8c61-86e50a8d2c78-kube-api-access-hqs89\") pod \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") "
Oct 30 01:01:37.875479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.859014    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-xks-ca\") pod \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") "
Oct 30 01:01:37.875479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.859167    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-openshift-ca\") pod \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\" (UID: \"09307ada-bbab-4ea2-8c61-86e50a8d2c78\") "
Oct 30 01:01:37.875479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.760868    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:01:37.875479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.862154    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"]
Oct 30 01:01:37.875479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.862943    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:01:37.875479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.862997    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:01:37.905483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.865050    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:37.905483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.876939    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqjx\" (UniqueName: \"kubernetes.io/projected/2d70886c-cc98-4c03-84f1-4cfdd6e447db-kube-api-access-6tqjx\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:37.905483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.877034    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:37.905483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.854578    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09307ada-bbab-4ea2-8c61-86e50a8d2c78-kube-api-access-hqs89" pod "09307ada-bbab-4ea2-8c61-86e50a8d2c78" (UID: "09307ada-bbab-4ea2-8c61-86e50a8d2c78"). InnerVolumeSpecName "kube-api-access-hqs89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:37.905483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.877864    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-openshift-ca" pod "09307ada-bbab-4ea2-8c61-86e50a8d2c78" (UID: "09307ada-bbab-4ea2-8c61-86e50a8d2c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:37.905483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.857972    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:37.905483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.791008    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:01:37.931480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.931409    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-xks-ca" pod "09307ada-bbab-4ea2-8c61-86e50a8d2c78" (UID: "09307ada-bbab-4ea2-8c61-86e50a8d2c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:37.986375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.986335    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:37.986742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.986717    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqjx\" (UniqueName: \"kubernetes.io/projected/2d70886c-cc98-4c03-84f1-4cfdd6e447db-kube-api-access-6tqjx\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:37.986927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.986910    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:37.987170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.987150    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqs89\" (UniqueName: \"kubernetes.io/projected/09307ada-bbab-4ea2-8c61-86e50a8d2c78-kube-api-access-hqs89\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:37.987276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.987262    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:37.987384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.987369    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09307ada-bbab-4ea2-8c61-86e50a8d2c78-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:37.992007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:37.991983    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:38.025435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.025327    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:38.034409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.032238    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqjx\" (UniqueName: \"kubernetes.io/projected/2d70886c-cc98-4c03-84f1-4cfdd6e447db-kube-api-access-6tqjx\") pod \"maas-api-revoke-sub-4d512610-kfzn7\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:38.090157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.089450    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-xks-ca\") pod \"cbf18d57-9466-4bc6-a645-08d241c4061f\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") "
Oct 30 01:01:38.090157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.089667    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf18d57-9466-4bc6-a645-08d241c4061f-kube-api-access-sgn4v\" (UniqueName: \"kubernetes.io/projected/cbf18d57-9466-4bc6-a645-08d241c4061f-kube-api-access-sgn4v\") pod \"cbf18d57-9466-4bc6-a645-08d241c4061f\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") "
Oct 30 01:01:38.090157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.089719    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-openshift-ca\") pod \"cbf18d57-9466-4bc6-a645-08d241c4061f\" (UID: \"cbf18d57-9466-4bc6-a645-08d241c4061f\") "
Oct 30 01:01:38.094605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.094428    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-xks-ca" pod "cbf18d57-9466-4bc6-a645-08d241c4061f" (UID: "cbf18d57-9466-4bc6-a645-08d241c4061f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:38.097497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.097440    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-openshift-ca" pod "cbf18d57-9466-4bc6-a645-08d241c4061f" (UID: "cbf18d57-9466-4bc6-a645-08d241c4061f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:38.104115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.104022    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf18d57-9466-4bc6-a645-08d241c4061f-kube-api-access-sgn4v" pod "cbf18d57-9466-4bc6-a645-08d241c4061f" (UID: "cbf18d57-9466-4bc6-a645-08d241c4061f"). InnerVolumeSpecName "kube-api-access-sgn4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:38.192430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.192394    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:38.192430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.192431    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgn4v\" (UniqueName: \"kubernetes.io/projected/cbf18d57-9466-4bc6-a645-08d241c4061f-kube-api-access-sgn4v\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:38.192663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.192446    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf18d57-9466-4bc6-a645-08d241c4061f-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:38.242992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.242962    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:38.317312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.316584    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"
Oct 30 01:01:38.317312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.316706    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg" event={"ID":"09307ada-bbab-4ea2-8c61-86e50a8d2c78","Type":"ContainerDied","Data":"ad1444a123a2e5093337f45ccb758da9da81ef037f479b2783aae5f18ef8fe78"}
Oct 30 01:01:38.317312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.316749    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad1444a123a2e5093337f45ccb758da9da81ef037f479b2783aae5f18ef8fe78"
Oct 30 01:01:38.327028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.326992    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"
Oct 30 01:01:38.327028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.327021    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k" event={"ID":"cbf18d57-9466-4bc6-a645-08d241c4061f","Type":"ContainerDied","Data":"aee73bc1ca65a2da758fdc07165ec455f7e9d1027b98c0779d2ca3774309fbc8"}
Oct 30 01:01:38.327252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.327070    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee73bc1ca65a2da758fdc07165ec455f7e9d1027b98c0779d2ca3774309fbc8"
Oct 30 01:01:38.548146 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:38.548112    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d70886c_cc98_4c03_84f1_4cfdd6e447db.slice/crio-10e477465d58c63978f0c2e663378565e884146f58b1bad1d700777c49daae1a WatchSource:0}: Error finding container 10e477465d58c63978f0c2e663378565e884146f58b1bad1d700777c49daae1a: Status 404 returned error can't find the container with id 10e477465d58c63978f0c2e663378565e884146f58b1bad1d700777c49daae1a
Oct 30 01:01:38.549255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:38.549233    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"]
Oct 30 01:01:39.113070 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:01:39.113022    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:01:39.338059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.338019    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7" event={"ID":"2d70886c-cc98-4c03-84f1-4cfdd6e447db","Type":"ContainerStarted","Data":"84ffc2d62c400a1c40664c33675ca73202807161895706f8162a17accabd20f6"}
Oct 30 01:01:39.338274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.338086    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7" event={"ID":"2d70886c-cc98-4c03-84f1-4cfdd6e447db","Type":"ContainerStarted","Data":"10e477465d58c63978f0c2e663378565e884146f58b1bad1d700777c49daae1a"}
Oct 30 01:01:39.348018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.347991    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"]
Oct 30 01:01:39.348681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.348667    5104 cpu_manager.go:414] "containerMap: removing container" podUID="cbf18d57-9466-4bc6-a645-08d241c4061f" containerName="revoke-keys"
Oct 30 01:01:39.348731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.348683    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf18d57-9466-4bc6-a645-08d241c4061f" containerName="revoke-keys"
Oct 30 01:01:39.348731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.348706    5104 cpu_manager.go:414] "containerMap: removing container" podUID="09307ada-bbab-4ea2-8c61-86e50a8d2c78" containerName="revoke-keys"
Oct 30 01:01:39.348731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.348711    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09307ada-bbab-4ea2-8c61-86e50a8d2c78" containerName="revoke-keys"
Oct 30 01:01:39.348838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.348829    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf18d57-9466-4bc6-a645-08d241c4061f" containerName="revoke-keys"
Oct 30 01:01:39.348872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.348842    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="09307ada-bbab-4ea2-8c61-86e50a8d2c78" containerName="revoke-keys"
Oct 30 01:01:39.352954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.352935    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.371653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.371620    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"]
Oct 30 01:01:39.381644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.381588    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7" podStartSLOduration=5.381570696 podStartE2EDuration="5.381570696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:39.374554423 +0000 UTC m=+1140.922244930" watchObservedRunningTime="2026-10-30 01:01:39.381570696 +0000 UTC m=+1140.929261191"
Oct 30 01:01:39.426838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.426807    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2qm\" (UniqueName: \"kubernetes.io/projected/7c081df0-12e0-4e96-864b-60f9aa88a732-kube-api-access-5d2qm\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.427008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.426955    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.427250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.427215    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.528403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.528366    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.528641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.528621    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.528693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.528679    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2qm\" (UniqueName: \"kubernetes.io/projected/7c081df0-12e0-4e96-864b-60f9aa88a732-kube-api-access-5d2qm\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.529317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.529285    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.530910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.530886    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.546987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.546961    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2qm\" (UniqueName: \"kubernetes.io/projected/7c081df0-12e0-4e96-864b-60f9aa88a732-kube-api-access-5d2qm\") pod \"maas-api-revoke-sub-3ba82ac8-jxmp7\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.667559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.667523    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:39.971285 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:39.971253    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c081df0_12e0_4e96_864b_60f9aa88a732.slice/crio-75d54503adf7cd733ade072363d90aedf61006a60e91e786712795166ed20eed WatchSource:0}: Error finding container 75d54503adf7cd733ade072363d90aedf61006a60e91e786712795166ed20eed: Status 404 returned error can't find the container with id 75d54503adf7cd733ade072363d90aedf61006a60e91e786712795166ed20eed
Oct 30 01:01:39.973176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:39.973148    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"]
Oct 30 01:01:40.347829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:40.347743    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7" event={"ID":"7c081df0-12e0-4e96-864b-60f9aa88a732","Type":"ContainerStarted","Data":"75d54503adf7cd733ade072363d90aedf61006a60e91e786712795166ed20eed"}
Oct 30 01:01:40.349546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:40.349518    5104 generic.go:356] "Generic (PLEG): container finished" podID="2d70886c-cc98-4c03-84f1-4cfdd6e447db" containerID="84ffc2d62c400a1c40664c33675ca73202807161895706f8162a17accabd20f6" exitCode=0
Oct 30 01:01:40.349680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:40.349607    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7" event={"ID":"2d70886c-cc98-4c03-84f1-4cfdd6e447db","Type":"ContainerDied","Data":"84ffc2d62c400a1c40664c33675ca73202807161895706f8162a17accabd20f6"}
Oct 30 01:01:41.361691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.361604    5104 generic.go:356] "Generic (PLEG): container finished" podID="7c081df0-12e0-4e96-864b-60f9aa88a732" containerID="c29633b0dd7b301e394330f34ae73909daceec236b3ec8cddd8832eeedbd84a7" exitCode=0
Oct 30 01:01:41.362080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.361696    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7" event={"ID":"7c081df0-12e0-4e96-864b-60f9aa88a732","Type":"ContainerDied","Data":"c29633b0dd7b301e394330f34ae73909daceec236b3ec8cddd8832eeedbd84a7"}
Oct 30 01:01:41.544640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.544608    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:41.568646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.568616    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-xks-ca\") pod \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") "
Oct 30 01:01:41.568866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.568668    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d70886c-cc98-4c03-84f1-4cfdd6e447db-kube-api-access-6tqjx\" (UniqueName: \"kubernetes.io/projected/2d70886c-cc98-4c03-84f1-4cfdd6e447db-kube-api-access-6tqjx\") pod \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") "
Oct 30 01:01:41.568932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.568892    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-openshift-ca\") pod \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\" (UID: \"2d70886c-cc98-4c03-84f1-4cfdd6e447db\") "
Oct 30 01:01:41.569240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.569205    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-openshift-ca" pod "2d70886c-cc98-4c03-84f1-4cfdd6e447db" (UID: "2d70886c-cc98-4c03-84f1-4cfdd6e447db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:41.569394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.569376    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:41.570912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.570887    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-xks-ca" pod "2d70886c-cc98-4c03-84f1-4cfdd6e447db" (UID: "2d70886c-cc98-4c03-84f1-4cfdd6e447db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:41.571018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.570897    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d70886c-cc98-4c03-84f1-4cfdd6e447db-kube-api-access-6tqjx" pod "2d70886c-cc98-4c03-84f1-4cfdd6e447db" (UID: "2d70886c-cc98-4c03-84f1-4cfdd6e447db"). InnerVolumeSpecName "kube-api-access-6tqjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:41.670882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.670843    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d70886c-cc98-4c03-84f1-4cfdd6e447db-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:41.670882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:41.670877    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tqjx\" (UniqueName: \"kubernetes.io/projected/2d70886c-cc98-4c03-84f1-4cfdd6e447db-kube-api-access-6tqjx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:42.405331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.405292    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"
Oct 30 01:01:42.405331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.405312    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7" event={"ID":"2d70886c-cc98-4c03-84f1-4cfdd6e447db","Type":"ContainerDied","Data":"10e477465d58c63978f0c2e663378565e884146f58b1bad1d700777c49daae1a"}
Oct 30 01:01:42.405896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.405369    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e477465d58c63978f0c2e663378565e884146f58b1bad1d700777c49daae1a"
Oct 30 01:01:42.605414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.605384    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:42.700476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.695165    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-xks-ca\") pod \"7c081df0-12e0-4e96-864b-60f9aa88a732\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") "
Oct 30 01:01:42.700476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.695309    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c081df0-12e0-4e96-864b-60f9aa88a732-kube-api-access-5d2qm\" (UniqueName: \"kubernetes.io/projected/7c081df0-12e0-4e96-864b-60f9aa88a732-kube-api-access-5d2qm\") pod \"7c081df0-12e0-4e96-864b-60f9aa88a732\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") "
Oct 30 01:01:42.700476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.695369    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-openshift-ca\") pod \"7c081df0-12e0-4e96-864b-60f9aa88a732\" (UID: \"7c081df0-12e0-4e96-864b-60f9aa88a732\") "
Oct 30 01:01:42.700476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.696477    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-openshift-ca" pod "7c081df0-12e0-4e96-864b-60f9aa88a732" (UID: "7c081df0-12e0-4e96-864b-60f9aa88a732"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:42.708115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.701156    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c081df0-12e0-4e96-864b-60f9aa88a732-kube-api-access-5d2qm" pod "7c081df0-12e0-4e96-864b-60f9aa88a732" (UID: "7c081df0-12e0-4e96-864b-60f9aa88a732"). InnerVolumeSpecName "kube-api-access-5d2qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:42.708115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.701532    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-xks-ca" pod "7c081df0-12e0-4e96-864b-60f9aa88a732" (UID: "7c081df0-12e0-4e96-864b-60f9aa88a732"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:42.797771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.797740    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:42.797771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.797767    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d2qm\" (UniqueName: \"kubernetes.io/projected/7c081df0-12e0-4e96-864b-60f9aa88a732-kube-api-access-5d2qm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:42.798036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:42.797778    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c081df0-12e0-4e96-864b-60f9aa88a732-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:43.417346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:43.417315    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"
Oct 30 01:01:43.417346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:43.417338    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7" event={"ID":"7c081df0-12e0-4e96-864b-60f9aa88a732","Type":"ContainerDied","Data":"75d54503adf7cd733ade072363d90aedf61006a60e91e786712795166ed20eed"}
Oct 30 01:01:43.417852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:43.417389    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d54503adf7cd733ade072363d90aedf61006a60e91e786712795166ed20eed"
Oct 30 01:01:48.005112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.005072    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"]
Oct 30 01:01:48.006358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.006331    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2d70886c-cc98-4c03-84f1-4cfdd6e447db" containerName="revoke-keys"
Oct 30 01:01:48.006358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.006359    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d70886c-cc98-4c03-84f1-4cfdd6e447db" containerName="revoke-keys"
Oct 30 01:01:48.006524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.006445    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7c081df0-12e0-4e96-864b-60f9aa88a732" containerName="revoke-keys"
Oct 30 01:01:48.006524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.006471    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c081df0-12e0-4e96-864b-60f9aa88a732" containerName="revoke-keys"
Oct 30 01:01:48.007103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.006661    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c081df0-12e0-4e96-864b-60f9aa88a732" containerName="revoke-keys"
Oct 30 01:01:48.007103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.006682    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d70886c-cc98-4c03-84f1-4cfdd6e447db" containerName="revoke-keys"
Oct 30 01:01:48.013226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.013200    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.045253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.045220    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"]
Oct 30 01:01:48.047770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.047747    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0660f91f-openshift-default-dockercfg-wl72z\""
Oct 30 01:01:48.104932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.104890    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-credential-socket\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105035    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-envoy\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105092    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-socket\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105151    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istiod-ca-cert\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105273    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-token\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkcc\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-kube-api-access-pmkcc\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105406    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-data\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105473    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-podinfo\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.105536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.105502    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-certs\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.252559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.209695    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-credential-socket\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.252559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.209784    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-envoy\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.253373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.253327    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-credential-socket\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.254884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.254857    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-socket\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.255077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.255027    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-socket\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.255364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.255310    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istiod-ca-cert\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.255611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.255580    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-token\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.265478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.255789    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkcc\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-kube-api-access-pmkcc\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.265785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.258143    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istiod-ca-cert\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.265994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.265959    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-data\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.266197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.266165    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-podinfo\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.266340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.266325    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-certs\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.267070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.267045    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-certs\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.278568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.269805    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-data\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.294100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.294028    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-token\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.297152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.297091    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkcc\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-kube-api-access-pmkcc\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.320733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.303710    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"]
Oct 30 01:01:48.320733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.317943    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-envoy\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.382477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.363145    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-podinfo\") pod \"e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") " pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.403656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.402818    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"]
Oct 30 01:01:48.403656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.402971    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.423761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.417714    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:48.423761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.418514    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:48.441523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.438029    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:48.441523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.438174    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:01:48.551769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.503571    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.551769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.522034    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.553102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.553070    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwml4\" (UniqueName: \"kubernetes.io/projected/2e185f0b-b201-4239-b7c6-a7a00374f957-kube-api-access-pwml4\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.653645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.653614    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:48.655379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.655350    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.655526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.655406    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.655605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.655532    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwml4\" (UniqueName: \"kubernetes.io/projected/2e185f0b-b201-4239-b7c6-a7a00374f957-kube-api-access-pwml4\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.656271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.656243    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.657988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.657965    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.679569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.679517    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwml4\" (UniqueName: \"kubernetes.io/projected/2e185f0b-b201-4239-b7c6-a7a00374f957-kube-api-access-pwml4\") pod \"maas-api-revoke-sub-16cd7001-2gskp\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.860284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.860189    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:48.983121 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:48.983079    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae55dd8_21c4_402f_80b9_0d9d91ff9cd6.slice/crio-2da29f0d080b51b446acba390496cdc52b0c8f26a92de37d26d27af974f89bd6 WatchSource:0}: Error finding container 2da29f0d080b51b446acba390496cdc52b0c8f26a92de37d26d27af974f89bd6: Status 404 returned error can't find the container with id 2da29f0d080b51b446acba390496cdc52b0c8f26a92de37d26d27af974f89bd6
Oct 30 01:01:48.984032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.984003    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"]
Oct 30 01:01:48.986234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.986198    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:48.986345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.986290    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:48.986345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:48.986342    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:49.184503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.179486    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"]
Oct 30 01:01:49.603274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.603237    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" event={"ID":"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6","Type":"ContainerStarted","Data":"961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba"}
Oct 30 01:01:49.603425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.603300    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" event={"ID":"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6","Type":"ContainerStarted","Data":"2da29f0d080b51b446acba390496cdc52b0c8f26a92de37d26d27af974f89bd6"}
Oct 30 01:01:49.605184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.605159    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp" event={"ID":"2e185f0b-b201-4239-b7c6-a7a00374f957","Type":"ContainerStarted","Data":"a753de4e01f2296c14990c34fedeb392e817dc1838d9212f7f062aa4a6a2f311"}
Oct 30 01:01:49.605292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.605203    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp" event={"ID":"2e185f0b-b201-4239-b7c6-a7a00374f957","Type":"ContainerStarted","Data":"2a17bd0b3715d4bbc1e2ead9b0fd4dbe40c672a2b1b7baae3a4e735e98d47ef5"}
Oct 30 01:01:49.650388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.650338    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podStartSLOduration=2.65032412 podStartE2EDuration="2.65032412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:49.644635836 +0000 UTC m=+1151.192326331" watchObservedRunningTime="2026-10-30 01:01:49.65032412 +0000 UTC m=+1151.198014615"
Oct 30 01:01:49.654406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.654382    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:49.685385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:49.685341    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp" podStartSLOduration=1.685328351 podStartE2EDuration="1.685328351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:49.680832465 +0000 UTC m=+1151.228522961" watchObservedRunningTime="2026-10-30 01:01:49.685328351 +0000 UTC m=+1151.233018847"
Oct 30 01:01:50.617190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:50.617155    5104 generic.go:356] "Generic (PLEG): container finished" podID="2e185f0b-b201-4239-b7c6-a7a00374f957" containerID="a753de4e01f2296c14990c34fedeb392e817dc1838d9212f7f062aa4a6a2f311" exitCode=0
Oct 30 01:01:50.617631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:50.617193    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp" event={"ID":"2e185f0b-b201-4239-b7c6-a7a00374f957","Type":"ContainerDied","Data":"a753de4e01f2296c14990c34fedeb392e817dc1838d9212f7f062aa4a6a2f311"}
Oct 30 01:01:50.659887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:50.659859    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:51.627911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.627865    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:51.629084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.629066    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:01:51.847292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.847267    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:51.937158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.937125    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-xks-ca\") pod \"2e185f0b-b201-4239-b7c6-a7a00374f957\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") "
Oct 30 01:01:51.937358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.937223    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-openshift-ca\") pod \"2e185f0b-b201-4239-b7c6-a7a00374f957\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") "
Oct 30 01:01:51.937358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.937256    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e185f0b-b201-4239-b7c6-a7a00374f957-kube-api-access-pwml4\" (UniqueName: \"kubernetes.io/projected/2e185f0b-b201-4239-b7c6-a7a00374f957-kube-api-access-pwml4\") pod \"2e185f0b-b201-4239-b7c6-a7a00374f957\" (UID: \"2e185f0b-b201-4239-b7c6-a7a00374f957\") "
Oct 30 01:01:51.937706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.937672    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-openshift-ca" pod "2e185f0b-b201-4239-b7c6-a7a00374f957" (UID: "2e185f0b-b201-4239-b7c6-a7a00374f957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:51.937998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.937971    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:51.939504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.939442    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-xks-ca" pod "2e185f0b-b201-4239-b7c6-a7a00374f957" (UID: "2e185f0b-b201-4239-b7c6-a7a00374f957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:51.939689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:51.939668    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e185f0b-b201-4239-b7c6-a7a00374f957-kube-api-access-pwml4" pod "2e185f0b-b201-4239-b7c6-a7a00374f957" (UID: "2e185f0b-b201-4239-b7c6-a7a00374f957"). InnerVolumeSpecName "kube-api-access-pwml4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:52.039179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.039146    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwml4\" (UniqueName: \"kubernetes.io/projected/2e185f0b-b201-4239-b7c6-a7a00374f957-kube-api-access-pwml4\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:52.039179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.039177    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e185f0b-b201-4239-b7c6-a7a00374f957-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:01:52.490115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.490079    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"]
Oct 30 01:01:52.492079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.491293    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2e185f0b-b201-4239-b7c6-a7a00374f957" containerName="revoke-keys"
Oct 30 01:01:52.492079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.491318    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e185f0b-b201-4239-b7c6-a7a00374f957" containerName="revoke-keys"
Oct 30 01:01:52.492079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.491563    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e185f0b-b201-4239-b7c6-a7a00374f957" containerName="revoke-keys"
Oct 30 01:01:52.496165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.496143    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.515206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.515175    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"]
Oct 30 01:01:52.519835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.519807    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8b58192f-openshift-default-dockercfg-cl778\""
Oct 30 01:01:52.560732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.560492    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d47ab9-6401-40f8-9533-756429e691f9-istio-podinfo\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.560732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.560606    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-credential-socket\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.560732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.560673    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-socket\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.560732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.560728    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-data\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.561082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.560765    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-istio-token\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.561082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.560855    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvslf\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-kube-api-access-nvslf\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.561082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.560902    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-certs\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.561082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.561046    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d47ab9-6401-40f8-9533-756429e691f9-istiod-ca-cert\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.561281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.561104    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-envoy\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678326    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d47ab9-6401-40f8-9533-756429e691f9-istiod-ca-cert\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678434    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-envoy\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678534    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d47ab9-6401-40f8-9533-756429e691f9-istio-podinfo\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678650    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-credential-socket\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678717    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-socket\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678782    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-data\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678831    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-istio-token\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678915    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvslf\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-kube-api-access-nvslf\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.678969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-certs\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.680069    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-data\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.680373    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-credential-socket\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.680644    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-socket\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.682096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.681193    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d47ab9-6401-40f8-9533-756429e691f9-istiod-ca-cert\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.683411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.683389    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-certs\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.704530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.704488    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"
Oct 30 01:01:52.730828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.705614    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d47ab9-6401-40f8-9533-756429e691f9-istio-podinfo\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.730828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.705651    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp" event={"ID":"2e185f0b-b201-4239-b7c6-a7a00374f957","Type":"ContainerDied","Data":"2a17bd0b3715d4bbc1e2ead9b0fd4dbe40c672a2b1b7baae3a4e735e98d47ef5"}
Oct 30 01:01:52.730828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.730735    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a17bd0b3715d4bbc1e2ead9b0fd4dbe40c672a2b1b7baae3a4e735e98d47ef5"
Oct 30 01:01:52.730828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.712310    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvslf\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-kube-api-access-nvslf\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.746643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.736071    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-istio-token\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.746886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.746798    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-envoy\") pod \"e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") " pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:52.820250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:52.820212    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:53.216474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.216431    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"]
Oct 30 01:01:53.222624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.222601    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.236224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.236188    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0660f91f-dockercfg-dcrms\""
Oct 30 01:01:53.237763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.236308    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0660f91f\""
Oct 30 01:01:53.241076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.241050    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"]
Oct 30 01:01:53.258887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.258858    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"]
Oct 30 01:01:53.259536 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:53.259509    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d47ab9_6401_40f8_9533_756429e691f9.slice/crio-60186ad79476c6ddee3805319a0bee35b0d6f677dfdf011c8aac44ecf306989c WatchSource:0}: Error finding container 60186ad79476c6ddee3805319a0bee35b0d6f677dfdf011c8aac44ecf306989c: Status 404 returned error can't find the container with id 60186ad79476c6ddee3805319a0bee35b0d6f677dfdf011c8aac44ecf306989c
Oct 30 01:01:53.274100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.273686    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:53.274100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.273771    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:53.274100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.273813    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:01:53.375297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.374886    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c23ee9-4928-4167-94c3-8c56160b186d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.375297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.375169    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vtw\" (UniqueName: \"kubernetes.io/projected/21c23ee9-4928-4167-94c3-8c56160b186d-kube-api-access-k2vtw\") pod \"payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.438481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.426717    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"]
Oct 30 01:01:53.471082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.471053    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"]
Oct 30 01:01:53.471378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.471361    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:53.519732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.516245    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c23ee9-4928-4167-94c3-8c56160b186d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.519732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.516417    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae50783-4723-47c1-ba49-e85fd84a7321-plugins-config-volume\") pod \"payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") " pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:53.519732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.516477    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jgh\" (UniqueName: \"kubernetes.io/projected/8ae50783-4723-47c1-ba49-e85fd84a7321-kube-api-access-79jgh\") pod \"payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") " pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:53.519732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.516581    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vtw\" (UniqueName: \"kubernetes.io/projected/21c23ee9-4928-4167-94c3-8c56160b186d-kube-api-access-k2vtw\") pod \"payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.575396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.575359    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c23ee9-4928-4167-94c3-8c56160b186d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.620336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.620297    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae50783-4723-47c1-ba49-e85fd84a7321-plugins-config-volume\") pod \"payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") " pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:53.633606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.633566    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jgh\" (UniqueName: \"kubernetes.io/projected/8ae50783-4723-47c1-ba49-e85fd84a7321-kube-api-access-79jgh\") pod \"payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") " pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:53.656407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.646148    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae50783-4723-47c1-ba49-e85fd84a7321-plugins-config-volume\") pod \"payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") " pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:53.656407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.651224    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vtw\" (UniqueName: \"kubernetes.io/projected/21c23ee9-4928-4167-94c3-8c56160b186d-kube-api-access-k2vtw\") pod \"payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.690379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.688558    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jgh\" (UniqueName: \"kubernetes.io/projected/8ae50783-4723-47c1-ba49-e85fd84a7321-kube-api-access-79jgh\") pod \"payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") " pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:53.720696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.719371    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78" event={"ID":"52d47ab9-6401-40f8-9533-756429e691f9","Type":"ContainerStarted","Data":"ed1a4debad22adba3bf694a476ef9d3ff38780964e624ca0263ec5a2182962bb"}
Oct 30 01:01:53.720696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.719429    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78" event={"ID":"52d47ab9-6401-40f8-9533-756429e691f9","Type":"ContainerStarted","Data":"60186ad79476c6ddee3805319a0bee35b0d6f677dfdf011c8aac44ecf306989c"}
Oct 30 01:01:53.769038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.768968    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78" podStartSLOduration=1.7689463 podStartE2EDuration="1.7689463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:53.760238394 +0000 UTC m=+1155.307928890" watchObservedRunningTime="2026-10-30 01:01:53.7689463 +0000 UTC m=+1155.316636796"
Oct 30 01:01:53.822359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.822227    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:53.841648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.841619    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:53.870577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.870545    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"]
Oct 30 01:01:53.880500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.877563    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:53.917684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.894295    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:53.917684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.895028    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:01:53.917684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.895082    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:53.917684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.895142    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:53.949481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:53.926586    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:54.079334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.079242    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.100630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.092847    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzv9d\" (UniqueName: \"kubernetes.io/projected/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-kube-api-access-gzv9d\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.105397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.105360    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.143745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.130146    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"]
Oct 30 01:01:54.207269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.207195    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.207402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.207347    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.207669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.207649    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzv9d\" (UniqueName: \"kubernetes.io/projected/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-kube-api-access-gzv9d\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.208347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.208294    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.210297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.210269    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.226200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.226170    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzv9d\" (UniqueName: \"kubernetes.io/projected/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-kube-api-access-gzv9d\") pod \"maas-api-revoke-sub-bfac88a7-xj7mh\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.398710 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:54.398668    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c23ee9_4928_4167_94c3_8c56160b186d.slice/crio-545551ce784181d523f2596160e16e1c165b4e734ad2c46a1bcc83934477309d WatchSource:0}: Error finding container 545551ce784181d523f2596160e16e1c165b4e734ad2c46a1bcc83934477309d: Status 404 returned error can't find the container with id 545551ce784181d523f2596160e16e1c165b4e734ad2c46a1bcc83934477309d
Oct 30 01:01:54.401530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.401504    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"]
Oct 30 01:01:54.424143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.424106    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:01:54.442894 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:54.442858    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae50783_4723_47c1_ba49_e85fd84a7321.slice/crio-020edfb6ded6b9ec70b5ac3ed6bf04609be54faed8aa608f9075d64448a2beba WatchSource:0}: Error finding container 020edfb6ded6b9ec70b5ac3ed6bf04609be54faed8aa608f9075d64448a2beba: Status 404 returned error can't find the container with id 020edfb6ded6b9ec70b5ac3ed6bf04609be54faed8aa608f9075d64448a2beba
Oct 30 01:01:54.445147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.445125    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"]
Oct 30 01:01:54.728282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.728243    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw" event={"ID":"8ae50783-4723-47c1-ba49-e85fd84a7321","Type":"ContainerStarted","Data":"064e914c837bb8ceb53f4c68b0fe88ed16f495e2457123a96ca376145a74afeb"}
Oct 30 01:01:54.728754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.728299    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw" event={"ID":"8ae50783-4723-47c1-ba49-e85fd84a7321","Type":"ContainerStarted","Data":"020edfb6ded6b9ec70b5ac3ed6bf04609be54faed8aa608f9075d64448a2beba"}
Oct 30 01:01:54.728754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.728345    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:01:54.730220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.730187    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt" event={"ID":"21c23ee9-4928-4167-94c3-8c56160b186d","Type":"ContainerStarted","Data":"11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0"}
Oct 30 01:01:54.730372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.730274    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt" event={"ID":"21c23ee9-4928-4167-94c3-8c56160b186d","Type":"ContainerStarted","Data":"545551ce784181d523f2596160e16e1c165b4e734ad2c46a1bcc83934477309d"}
Oct 30 01:01:54.742155 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:01:54.742132    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1058d6ce_bd7b_4fdc_8e20_1272a92d3c38.slice/crio-81d07a2cd9a22b406e91e072020a76d146d483b8feaaf8d6f020f77a2ff59546 WatchSource:0}: Error finding container 81d07a2cd9a22b406e91e072020a76d146d483b8feaaf8d6f020f77a2ff59546: Status 404 returned error can't find the container with id 81d07a2cd9a22b406e91e072020a76d146d483b8feaaf8d6f020f77a2ff59546
Oct 30 01:01:54.743528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.743507    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"]
Oct 30 01:01:54.770236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.770183    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw" podStartSLOduration=1.770168859 podStartE2EDuration="1.770168859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:54.764848372 +0000 UTC m=+1156.312538869" watchObservedRunningTime="2026-10-30 01:01:54.770168859 +0000 UTC m=+1156.317859355"
Oct 30 01:01:54.806048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.805986    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt" podStartSLOduration=1.805965977 podStartE2EDuration="1.805965977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:54.801926605 +0000 UTC m=+1156.349617098" watchObservedRunningTime="2026-10-30 01:01:54.805965977 +0000 UTC m=+1156.353656467"
Oct 30 01:01:54.828782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:54.828752    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:55.649896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:55.649865    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:01:55.741734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:55.741689    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh" event={"ID":"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38","Type":"ContainerStarted","Data":"a3abe2b1138a443ed4e82e809172940e66b628319fc30a20e088908ed2ac71f7"}
Oct 30 01:01:55.742132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:55.741747    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh" event={"ID":"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38","Type":"ContainerStarted","Data":"81d07a2cd9a22b406e91e072020a76d146d483b8feaaf8d6f020f77a2ff59546"}
Oct 30 01:01:55.742132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:55.742102    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:01:55.742428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:55.742406    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:55.743385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:55.743364    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:01:55.785360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:55.785309    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh" podStartSLOduration=2.78529471 podStartE2EDuration="2.78529471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:55.78090158 +0000 UTC m=+1157.328592077" watchObservedRunningTime="2026-10-30 01:01:55.78529471 +0000 UTC m=+1157.332985206"
Oct 30 01:01:56.895476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:56.895255    5104 generic.go:356] "Generic (PLEG): container finished" podID="1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" containerID="a3abe2b1138a443ed4e82e809172940e66b628319fc30a20e088908ed2ac71f7" exitCode=0
Oct 30 01:01:56.907481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:56.898210    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh" event={"ID":"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38","Type":"ContainerDied","Data":"a3abe2b1138a443ed4e82e809172940e66b628319fc30a20e088908ed2ac71f7"}
Oct 30 01:01:57.698662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:57.698352    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:58.334481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:58.330342    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:01:58.381684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:58.379934    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:01:58.720630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:58.720567    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:58.729797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:58.724600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:01:58.795402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:58.776558    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:58.840700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:58.824633    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:01:59.008651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.008540    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:59.039943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.037562    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:59.060495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.054423    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:01:59.144741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.080050    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"]
Oct 30 01:01:59.218133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.218084    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"]
Oct 30 01:01:59.218509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.218487    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:01:59.304533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.263768    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8b58192f-dockercfg-5mw4v\""
Oct 30 01:01:59.304533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.301726    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8b58192f\""
Oct 30 01:01:59.334483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.330685    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:59.334483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.331277    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:01:59.396827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.386149    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc1b38-32cc-4853-824e-f3938c66e86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:01:59.396827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.386283    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jscx\" (UniqueName: \"kubernetes.io/projected/f8fc1b38-32cc-4853-824e-f3938c66e86e-kube-api-access-2jscx\") pod \"payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:01:59.488736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.484534    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"]
Oct 30 01:01:59.511001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.510962    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:59.561537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.523520    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jscx\" (UniqueName: \"kubernetes.io/projected/f8fc1b38-32cc-4853-824e-f3938c66e86e-kube-api-access-2jscx\") pod \"payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:01:59.561537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.523896    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc1b38-32cc-4853-824e-f3938c66e86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:01:59.561537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.525043    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc1b38-32cc-4853-824e-f3938c66e86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:01:59.665595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.654589    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"]
Oct 30 01:01:59.665595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.655018    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:01:59.726479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.726325    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:59.738986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.726941    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:59.738986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.727224    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:01:59.877863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.856502    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jscx\" (UniqueName: \"kubernetes.io/projected/f8fc1b38-32cc-4853-824e-f3938c66e86e-kube-api-access-2jscx\") pod \"payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:01:59.877863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.873838    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027a45ec-0dc6-4960-ae70-f66a54134342-plugins-config-volume\") pod \"payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") " pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:01:59.877863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:01:59.873976    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7s9\" (UniqueName: \"kubernetes.io/projected/027a45ec-0dc6-4960-ae70-f66a54134342-kube-api-access-mk7s9\") pod \"payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") " pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:00.030756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.030692    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:00.031965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.031522    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:00.152876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.031659    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:02:00.153198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.153177    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:02:00.153490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.153394    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:02:00.162060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.064002    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:00.162664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.162641    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:02:00.162836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.125434    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:00.163540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.126128    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:02:00.164176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.034547    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:00.174482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.063655    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7s9\" (UniqueName: \"kubernetes.io/projected/027a45ec-0dc6-4960-ae70-f66a54134342-kube-api-access-mk7s9\") pod \"payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") " pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:00.174482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.164962    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027a45ec-0dc6-4960-ae70-f66a54134342-plugins-config-volume\") pod \"payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") " pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:00.174482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.166012    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027a45ec-0dc6-4960-ae70-f66a54134342-plugins-config-volume\") pod \"payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") " pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:00.405706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.405600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:02:00.516688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.512307    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:00.630551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.628554    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:02:00.648500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.631023    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:02:00.785287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.785246    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:02:00.878215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.844574    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:00.878215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.844658    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:00.901627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.899644    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:00.901627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:00.899717    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:01.073963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.073866    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:02:01.176494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.175068    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:02:01.177474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.177423    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7s9\" (UniqueName: \"kubernetes.io/projected/027a45ec-0dc6-4960-ae70-f66a54134342-kube-api-access-mk7s9\") pod \"payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") " pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:01.177757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.177726    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:02:01.261551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.200664    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:02:01.261551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.201004    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:02:01.261551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.210654    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:02:01.416550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.276954    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:01.440481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.392260    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-xks-ca\") pod \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") "
Oct 30 01:02:01.440481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.440242    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-openshift-ca\") pod \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") "
Oct 30 01:02:01.440802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.440572    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-kube-api-access-gzv9d\" (UniqueName: \"kubernetes.io/projected/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-kube-api-access-gzv9d\") pod \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\" (UID: \"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38\") "
Oct 30 01:02:01.497475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.497412    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:01.664481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.660606    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:02:01.723483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.684863    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:01.723483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.688301    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-openshift-ca" pod "1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" (UID: "1058d6ce-bd7b-4fdc-8e20-1272a92d3c38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:01.726879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.725630    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:01.770635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.770584    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:01.810485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.803561    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:01.867503 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:01.838591    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.199s"
Oct 30 01:02:01.867503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.838679    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:02:01.867503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:01.838863    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:02:02.095559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.094159    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:02:02.185645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.185594    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:02:02.229261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.199411    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:02.520155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.288513    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:02.520155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.288551    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:02:02.520155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.316786    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-xks-ca" pod "1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" (UID: "1058d6ce-bd7b-4fdc-8e20-1272a92d3c38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:02.590389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.575415    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:02.590389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.586709    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:02.657149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.657111    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-kube-api-access-gzv9d" pod "1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" (UID: "1058d6ce-bd7b-4fdc-8e20-1272a92d3c38"). InnerVolumeSpecName "kube-api-access-gzv9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:02.690117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.690080    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzv9d\" (UniqueName: \"kubernetes.io/projected/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38-kube-api-access-gzv9d\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:02.702566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.702522    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:02.703982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.702792    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:02.748816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.748787    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"
Oct 30 01:02:02.948720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.908236    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:02:02.972481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.967375    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh" event={"ID":"1058d6ce-bd7b-4fdc-8e20-1272a92d3c38","Type":"ContainerDied","Data":"81d07a2cd9a22b406e91e072020a76d146d483b8feaaf8d6f020f77a2ff59546"}
Oct 30 01:02:02.972481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:02.967483    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d07a2cd9a22b406e91e072020a76d146d483b8feaaf8d6f020f77a2ff59546"
Oct 30 01:02:03.017877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.001771    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:03.017877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.002502    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:02:03.158534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.100218    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:03.196018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.195224    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:03.506623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.506570    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:02:03.542721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.542669    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:03.588093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.550616    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:03.630200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.590608    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:02:03.703727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:03.703670    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:04.043872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.043742    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:04.246810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.201841    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:04.250137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.239094    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:02:04.250429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.250412    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:02:04.250606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.250595    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:02:04.251928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.242310    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:04.304061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.303962    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:02:04.384192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.384154    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:02:04.432052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.431777    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:02:04.439318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.439282    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:02:04.477646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.477616    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:02:04.488517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.488484    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:02:04.507495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.504796    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:02:04.616799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.615648    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"]
Oct 30 01:02:04.620963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.620936    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" containerName="revoke-keys"
Oct 30 01:02:04.620963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.620963    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" containerName="revoke-keys"
Oct 30 01:02:04.621214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.621197    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" containerName="revoke-keys"
Oct 30 01:02:04.630205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.630169    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"]
Oct 30 01:02:04.630427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.630220    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"]
Oct 30 01:02:04.630427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.630314    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.648618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.648235    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:04.648618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.648317    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:04.673734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.673698    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.674014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.673992    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzmw\" (UniqueName: \"kubernetes.io/projected/c5ae3b79-af04-48db-a856-34ab58bd91bd-kube-api-access-8fzmw\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.674352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.674333    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.713933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.713882    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:02:04.714242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.714191    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:04.777563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.777528    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.778737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.778707    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.779861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.779671    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.780275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.779819    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzmw\" (UniqueName: \"kubernetes.io/projected/c5ae3b79-af04-48db-a856-34ab58bd91bd-kube-api-access-8fzmw\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.785409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.785390    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"]
Oct 30 01:02:04.814785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.814749    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.832813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.827290    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzmw\" (UniqueName: \"kubernetes.io/projected/c5ae3b79-af04-48db-a856-34ab58bd91bd-kube-api-access-8fzmw\") pod \"maas-api-revoke-sub-adacf3e4-tnbb5\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:04.953881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:04.953843    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:05.332065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.329503    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"]
Oct 30 01:02:05.629186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.629080    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:05.670879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.670849    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b61ed672-dockercfg-6fb6v\""
Oct 30 01:02:05.684060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.627903    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"]
Oct 30 01:02:05.684060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.677365    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"]
Oct 30 01:02:05.708337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.708292    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b61ed672\""
Oct 30 01:02:05.760795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.759898    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f565644-a0ed-430d-bc62-e3bac1584495-plugins-config-volume\") pod \"payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") " pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:05.760795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.760188    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9h54\" (UniqueName: \"kubernetes.io/projected/2f565644-a0ed-430d-bc62-e3bac1584495-kube-api-access-z9h54\") pod \"payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") " pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:05.761289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.761204    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5" event={"ID":"c5ae3b79-af04-48db-a856-34ab58bd91bd","Type":"ContainerStarted","Data":"6e9fe18534a667f4249cc0f26d56000dacce474bcb482b22e1fa5e1640791bf3"}
Oct 30 01:02:05.764608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.763662    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:02:05.776519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.765176    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf" event={"ID":"027a45ec-0dc6-4960-ae70-f66a54134342","Type":"ContainerStarted","Data":"b01a4334a66d49b920dde98ad3b24fc401f317554658d29b00ca5767c641a181"}
Oct 30 01:02:05.776519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.765223    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf" event={"ID":"027a45ec-0dc6-4960-ae70-f66a54134342","Type":"ContainerStarted","Data":"d791de638dce696ad554cac8bf0afb5ca67290bfb513add63592c775628fa6ae"}
Oct 30 01:02:05.777026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.776994    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:05.786358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.786327    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66" event={"ID":"f8fc1b38-32cc-4853-824e-f3938c66e86e","Type":"ContainerStarted","Data":"f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f"}
Oct 30 01:02:05.786517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.786383    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66" event={"ID":"f8fc1b38-32cc-4853-824e-f3938c66e86e","Type":"ContainerStarted","Data":"5eb8ea3c727a9879209b76495a736f51ece1e9d537912ec449c4c17a9cddd465"}
Oct 30 01:02:05.787515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.787495    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:02:05.857055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.856977    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66" podStartSLOduration=7.856961806 podStartE2EDuration="7.856961806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:05.84396705 +0000 UTC m=+1167.391657548" watchObservedRunningTime="2026-10-30 01:02:05.856961806 +0000 UTC m=+1167.404652302"
Oct 30 01:02:05.861349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.861316    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f565644-a0ed-430d-bc62-e3bac1584495-plugins-config-volume\") pod \"payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") " pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:05.861736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.861711    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9h54\" (UniqueName: \"kubernetes.io/projected/2f565644-a0ed-430d-bc62-e3bac1584495-kube-api-access-z9h54\") pod \"payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") " pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:05.862228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.862207    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f565644-a0ed-430d-bc62-e3bac1584495-plugins-config-volume\") pod \"payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") " pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:05.881539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:05.881473    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9h54\" (UniqueName: \"kubernetes.io/projected/2f565644-a0ed-430d-bc62-e3bac1584495-kube-api-access-z9h54\") pod \"payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") " pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:06.017777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.017737    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:06.422964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.422895    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf" podStartSLOduration=8.4228749 podStartE2EDuration="8.4228749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:05.882470741 +0000 UTC m=+1167.430161262" watchObservedRunningTime="2026-10-30 01:02:06.4228749 +0000 UTC m=+1167.970565397"
Oct 30 01:02:06.427812 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:02:06.427775    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f565644_a0ed_430d_bc62_e3bac1584495.slice/crio-351a2951a145a733ded57542f9ef71e160d86a673f7613ff83661871235596ca WatchSource:0}: Error finding container 351a2951a145a733ded57542f9ef71e160d86a673f7613ff83661871235596ca: Status 404 returned error can't find the container with id 351a2951a145a733ded57542f9ef71e160d86a673f7613ff83661871235596ca
Oct 30 01:02:06.430241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.430185    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"]
Oct 30 01:02:06.804134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.803706    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk" event={"ID":"2f565644-a0ed-430d-bc62-e3bac1584495","Type":"ContainerStarted","Data":"acd71c90717706703082ecb51a837a296a9272839d88c937513cf63711f766e2"}
Oct 30 01:02:06.804134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.803778    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk" event={"ID":"2f565644-a0ed-430d-bc62-e3bac1584495","Type":"ContainerStarted","Data":"351a2951a145a733ded57542f9ef71e160d86a673f7613ff83661871235596ca"}
Oct 30 01:02:06.804134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.803826    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:06.806157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.806127    5104 generic.go:356] "Generic (PLEG): container finished" podID="c5ae3b79-af04-48db-a856-34ab58bd91bd" containerID="bd25f3a3684aec8cc28b9a427b7e85f030ed9a9f80f98ba78312cd9ca3ad3f15" exitCode=0
Oct 30 01:02:06.806314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.806198    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5" event={"ID":"c5ae3b79-af04-48db-a856-34ab58bd91bd","Type":"ContainerDied","Data":"bd25f3a3684aec8cc28b9a427b7e85f030ed9a9f80f98ba78312cd9ca3ad3f15"}
Oct 30 01:02:06.845048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.844990    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk" podStartSLOduration=1.844975458 podStartE2EDuration="1.844975458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:06.839171814 +0000 UTC m=+1168.386862311" watchObservedRunningTime="2026-10-30 01:02:06.844975458 +0000 UTC m=+1168.392665953"
Oct 30 01:02:06.901471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:06.901432    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:02:08.020695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.020663    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:08.122410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.122325    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-openshift-ca\") pod \"c5ae3b79-af04-48db-a856-34ab58bd91bd\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") "
Oct 30 01:02:08.122410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.122370    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-xks-ca\") pod \"c5ae3b79-af04-48db-a856-34ab58bd91bd\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") "
Oct 30 01:02:08.122704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.122536    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ae3b79-af04-48db-a856-34ab58bd91bd-kube-api-access-8fzmw\" (UniqueName: \"kubernetes.io/projected/c5ae3b79-af04-48db-a856-34ab58bd91bd-kube-api-access-8fzmw\") pod \"c5ae3b79-af04-48db-a856-34ab58bd91bd\" (UID: \"c5ae3b79-af04-48db-a856-34ab58bd91bd\") "
Oct 30 01:02:08.122794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.122765    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-openshift-ca" pod "c5ae3b79-af04-48db-a856-34ab58bd91bd" (UID: "c5ae3b79-af04-48db-a856-34ab58bd91bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:08.123032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.123015    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:08.124913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.124890    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ae3b79-af04-48db-a856-34ab58bd91bd-kube-api-access-8fzmw" pod "c5ae3b79-af04-48db-a856-34ab58bd91bd" (UID: "c5ae3b79-af04-48db-a856-34ab58bd91bd"). InnerVolumeSpecName "kube-api-access-8fzmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:08.125018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.124992    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-xks-ca" pod "c5ae3b79-af04-48db-a856-34ab58bd91bd" (UID: "c5ae3b79-af04-48db-a856-34ab58bd91bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:08.225282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.225241    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ae3b79-af04-48db-a856-34ab58bd91bd-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:08.225282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.225283    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fzmw\" (UniqueName: \"kubernetes.io/projected/c5ae3b79-af04-48db-a856-34ab58bd91bd-kube-api-access-8fzmw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:08.875920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.875825    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5" event={"ID":"c5ae3b79-af04-48db-a856-34ab58bd91bd","Type":"ContainerDied","Data":"6e9fe18534a667f4249cc0f26d56000dacce474bcb482b22e1fa5e1640791bf3"}
Oct 30 01:02:08.875920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.875900    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9fe18534a667f4249cc0f26d56000dacce474bcb482b22e1fa5e1640791bf3"
Oct 30 01:02:08.876217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:08.875976    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"
Oct 30 01:02:09.106415 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:09.106379    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:02:17.820929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:17.820895    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:17.821433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:17.821390    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:02:17.821747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:17.821700    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:02:21.152608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.152569    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"]
Oct 30 01:02:21.153586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.153562    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c5ae3b79-af04-48db-a856-34ab58bd91bd" containerName="revoke-keys"
Oct 30 01:02:21.153586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.153585    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ae3b79-af04-48db-a856-34ab58bd91bd" containerName="revoke-keys"
Oct 30 01:02:21.153853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.153838    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ae3b79-af04-48db-a856-34ab58bd91bd" containerName="revoke-keys"
Oct 30 01:02:21.159769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.159748    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.228110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.228050    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"]
Oct 30 01:02:21.333642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.333611    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.333825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.333753    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.333870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.333820    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.333945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.333917    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.334012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.333999    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzl6r\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-kube-api-access-tzl6r\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.334081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.334066    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.334125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.334110    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.334189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.334168    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.334313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.334214    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8481aa-9575-4f53-a63c-52c5e7826750-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.435914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.435873    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.435924    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8481aa-9575-4f53-a63c-52c5e7826750-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436041    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436152    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436233    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436278    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436337    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzl6r\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-kube-api-access-tzl6r\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436358    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436397    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436421    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436473    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436641    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436889    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.436915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.436908    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8481aa-9575-4f53-a63c-52c5e7826750-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.438899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.438873    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.439130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.439110    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.457566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.457534    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.457699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.457621    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzl6r\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-kube-api-access-tzl6r\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.514319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.514279    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:21.554131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.554077    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"]
Oct 30 01:02:21.569820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.569775    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.596437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.595885    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-82b647-openshift-default-dockercfg-w4zsm\""
Oct 30 01:02:21.638545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.626378    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"]
Oct 30 01:02:21.751084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751049    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751148    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751185    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751217    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751278    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751299    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751382    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381b806e-98cd-499f-8f1d-e6166a3956d6-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.751746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.751495    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5txl\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-kube-api-access-d5txl\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.852803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.852761    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.852803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.852801    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381b806e-98cd-499f-8f1d-e6166a3956d6-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.852880    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5txl\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-kube-api-access-d5txl\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.852926    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853008    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853051    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853101    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853189    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853233    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853370    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853691    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.853782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853745    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.854021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.853998    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381b806e-98cd-499f-8f1d-e6166a3956d6-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.854186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.854165    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.855613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.855590    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.855736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.855712    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.870977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.870954    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5txl\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-kube-api-access-d5txl\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:21.871151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:21.871132    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:22.055531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.055495    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:22.132481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.064967    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:22.132481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.065108    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:22.132481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.065187    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:22.132481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.101021    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"]
Oct 30 01:02:22.174204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.174120    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" event={"ID":"6e8481aa-9575-4f53-a63c-52c5e7826750","Type":"ContainerStarted","Data":"b18966896e2fafda01466be08c7963dc61cc733ae851fa19bae2085ac00df27b"}
Oct 30 01:02:22.511779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.511147    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"]
Oct 30 01:02:22.513993 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:02:22.513958    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381b806e_98cd_499f_8f1d_e6166a3956d6.slice/crio-a1acdd6df34df0f8355db4091d49f683a36e357b0bcc60dea2cba364111387ec WatchSource:0}: Error finding container a1acdd6df34df0f8355db4091d49f683a36e357b0bcc60dea2cba364111387ec: Status 404 returned error can't find the container with id a1acdd6df34df0f8355db4091d49f683a36e357b0bcc60dea2cba364111387ec
Oct 30 01:02:22.516948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.516918    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:22.517029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.516987    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:22.517029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:22.517018    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:23.190810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:23.190739    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" event={"ID":"6e8481aa-9575-4f53-a63c-52c5e7826750","Type":"ContainerStarted","Data":"4ea87c5023e101032fff0f15e68d2300458e6bc42adee8cc39aff25d3af6e48a"}
Oct 30 01:02:23.192823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:23.192795    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" event={"ID":"381b806e-98cd-499f-8f1d-e6166a3956d6","Type":"ContainerStarted","Data":"53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b"}
Oct 30 01:02:23.192977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:23.192841    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" event={"ID":"381b806e-98cd-499f-8f1d-e6166a3956d6","Type":"ContainerStarted","Data":"a1acdd6df34df0f8355db4091d49f683a36e357b0bcc60dea2cba364111387ec"}
Oct 30 01:02:23.259273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:23.259214    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podStartSLOduration=2.25919186 podStartE2EDuration="2.25919186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:23.240082047 +0000 UTC m=+1184.787772543" watchObservedRunningTime="2026-10-30 01:02:23.25919186 +0000 UTC m=+1184.806882359"
Oct 30 01:02:23.517546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:23.516821    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:23.521482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:23.520595    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:23.521482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:23.520656    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:24.056594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:24.056193    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:24.057901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:24.057876    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:24.058000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:24.057920    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:24.520488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:24.514877    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:24.520488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:24.514948    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:25.056558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:25.056523    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:25.056754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:25.056572    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:25.517211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:25.517174    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:25.517548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:25.517518    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:26.059480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:26.058601    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:26.059480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:26.058659    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:26.544651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:26.544612    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:26.547323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:26.547286    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:27.058481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.057913    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:27.058481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.057983    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:27.313541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.313054    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:27.313541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.313129    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:27.440610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.440553    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:27.510545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.508562    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:27.520927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.520887    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:27.521212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.521184    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:27.525586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.525543    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:27.547306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.547260    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:27.580658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.580546    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:27.705598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.705552    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:27.793614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:27.793555    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.059482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.058914    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:28.059482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.058985    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:28.080681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.080633    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:02:28.181495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.180685    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.286621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.286564    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.451521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.442561    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.451521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.443359    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:02:28.538570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.513593    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.538570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.519756    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:02:28.538570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.521085    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:28.538570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.521136    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:28.538570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.528543    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.538570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.529816    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:02:28.620248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.569567    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.620248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.570019    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:02:28.748632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.747275    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podStartSLOduration=7.747247609 podStartE2EDuration="7.747247609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:23.318898329 +0000 UTC m=+1184.866588944" watchObservedRunningTime="2026-10-30 01:02:28.747247609 +0000 UTC m=+1190.294938105"
Oct 30 01:02:28.775522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.755704    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:02:28.775522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.772724    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:02:28.866510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.801165    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:28.866510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.804232    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:02:28.882491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:28.881913    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 30 01:02:29.229484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.200624    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:29.229484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.200697    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:29.229484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.221366    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:29.238554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.230092    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:02:29.514382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.514258    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:02:29.516732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.516677    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:02:29.601487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.597698    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:29.601487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.599248    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:29.771685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:29.771566    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:30.062700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.019105    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:02:30.086561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.076586    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:30.184892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.076681    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:30.192614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.192578    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:30.192869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.192839    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:30.253509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.253434    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:30.270481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.254133    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:02:30.300483 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:30.296090    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.521s"
Oct 30 01:02:30.300483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.296288    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:02:30.300483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.296328    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:02:30.300483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.296371    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:02:30.300483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.296408    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:02:30.300483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.296438    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:02:30.300483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.296488    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:02:30.547878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.538405    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:30.547878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.538496    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:30.547878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.544611    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:30.547878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.544965    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Oct 30 01:02:30.612392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.612341    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:30.950809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.612694    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:30.988108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.886418    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:30.990336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:30.988875    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:31.126859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.126820    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:31.128598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.128559    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:31.159696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.159660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:02:31.159964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.159950    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:02:31.160110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.160097    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:02:31.160222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.160212    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:02:31.274661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.274560    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:31.295075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.295029    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Oct 30 01:02:31.664544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.663702    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:31.664544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.663771    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:31.815480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.811724    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:02:32.586577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.821755    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:02:32.634402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:31.951272    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:02:32.634402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.023592    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:32.634402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.079443    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:32.634402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.285066    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:32.659482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.313637    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:32.659482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.640834    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:32.659482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.526136    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:32.659482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.526078    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:32.659482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.644910    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:32.770377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.770333    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:02:32.805484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.802054    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:32.859540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:32.841889    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:02:33.036242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.028574    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:02:33.092757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.080084    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:33.092757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.080149    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:33.471479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.470022    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:02:33.471479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.471190    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:02:33.498319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.498278    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"]
Oct 30 01:02:33.509738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.509702    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"]
Oct 30 01:02:33.509994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.509812    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.515963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.515554    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 01:02:33.515963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.515616    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 01:02:33.529511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.527077    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:33.529511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.527279    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:02:33.529511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.527351    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:33.529511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.528489    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:33.552740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.552163    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:02:33.651053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.651009    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.651530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.651502    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.651622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.651548    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzfmp\" (UniqueName: \"kubernetes.io/projected/9df6a07d-d841-42ef-883a-7868e6db2c06-kube-api-access-bzfmp\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.752807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.752722    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.752980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.752961    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.753116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.752995    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzfmp\" (UniqueName: \"kubernetes.io/projected/9df6a07d-d841-42ef-883a-7868e6db2c06-kube-api-access-bzfmp\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.753659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.753636    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.755191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.755168    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.771985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.771950    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzfmp\" (UniqueName: \"kubernetes.io/projected/9df6a07d-d841-42ef-883a-7868e6db2c06-kube-api-access-bzfmp\") pod \"maas-api-revoke-keys-e2e-ait-0660f91f-q49dq\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:33.856484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:33.855850    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:34.065546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:34.060916    5104 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 01:02:34.065546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:34.060980    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 01:02:34.448142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:34.447046    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"]
Oct 30 01:02:34.450239 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:02:34.450196    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df6a07d_d841_42ef_883a_7868e6db2c06.slice/crio-b94bd75cfddaad024fefa16f69de49bf7d33c358a3a1907a075d9928c84e808b WatchSource:0}: Error finding container b94bd75cfddaad024fefa16f69de49bf7d33c358a3a1907a075d9928c84e808b: Status 404 returned error can't find the container with id b94bd75cfddaad024fefa16f69de49bf7d33c358a3a1907a075d9928c84e808b
Oct 30 01:02:35.055271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.055239    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:35.055905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.055876    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:35.058182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.058156    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:02:35.062508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.062487    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:35.062834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.062809    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:35.063581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.063559    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:02:35.475509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.474787    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq" event={"ID":"9df6a07d-d841-42ef-883a-7868e6db2c06","Type":"ContainerStarted","Data":"96ac9371c1d399899035077fedcef07d70523766fb68c160391b4b0721b45cb0"}
Oct 30 01:02:35.475509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.474887    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq" event={"ID":"9df6a07d-d841-42ef-883a-7868e6db2c06","Type":"ContainerStarted","Data":"b94bd75cfddaad024fefa16f69de49bf7d33c358a3a1907a075d9928c84e808b"}
Oct 30 01:02:35.724475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:35.717931    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq" podStartSLOduration=3.717908151 podStartE2EDuration="3.717908151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:35.548314925 +0000 UTC m=+1197.096005422" watchObservedRunningTime="2026-10-30 01:02:35.717908151 +0000 UTC m=+1197.265598652"
Oct 30 01:02:36.488349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:36.488309    5104 generic.go:356] "Generic (PLEG): container finished" podID="9df6a07d-d841-42ef-883a-7868e6db2c06" containerID="96ac9371c1d399899035077fedcef07d70523766fb68c160391b4b0721b45cb0" exitCode=0
Oct 30 01:02:36.488922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:36.488548    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq" event={"ID":"9df6a07d-d841-42ef-883a-7868e6db2c06","Type":"ContainerDied","Data":"96ac9371c1d399899035077fedcef07d70523766fb68c160391b4b0721b45cb0"}
Oct 30 01:02:37.684762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.678318    5104 patch_prober.go:29] interesting pod/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:37.684762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.678409    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:37.807355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.807273    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:37.847162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.847116    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-openshift-ca\") pod \"9df6a07d-d841-42ef-883a-7868e6db2c06\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") "
Oct 30 01:02:37.847367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.847200    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-xks-ca\") pod \"9df6a07d-d841-42ef-883a-7868e6db2c06\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") "
Oct 30 01:02:37.847464 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.847437    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df6a07d-d841-42ef-883a-7868e6db2c06-kube-api-access-bzfmp\" (UniqueName: \"kubernetes.io/projected/9df6a07d-d841-42ef-883a-7868e6db2c06-kube-api-access-bzfmp\") pod \"9df6a07d-d841-42ef-883a-7868e6db2c06\" (UID: \"9df6a07d-d841-42ef-883a-7868e6db2c06\") "
Oct 30 01:02:37.850915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.849738    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-openshift-ca" pod "9df6a07d-d841-42ef-883a-7868e6db2c06" (UID: "9df6a07d-d841-42ef-883a-7868e6db2c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:37.852866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.852831    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-xks-ca" pod "9df6a07d-d841-42ef-883a-7868e6db2c06" (UID: "9df6a07d-d841-42ef-883a-7868e6db2c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:37.853199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.853167    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df6a07d-d841-42ef-883a-7868e6db2c06-kube-api-access-bzfmp" pod "9df6a07d-d841-42ef-883a-7868e6db2c06" (UID: "9df6a07d-d841-42ef-883a-7868e6db2c06"). InnerVolumeSpecName "kube-api-access-bzfmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:37.950299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.950262    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:37.950299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.950296    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df6a07d-d841-42ef-883a-7868e6db2c06-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:37.950299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:37.950307    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzfmp\" (UniqueName: \"kubernetes.io/projected/9df6a07d-d841-42ef-883a-7868e6db2c06-kube-api-access-bzfmp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:38.535946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:38.535911    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"
Oct 30 01:02:38.536159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:38.535945    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq" event={"ID":"9df6a07d-d841-42ef-883a-7868e6db2c06","Type":"ContainerDied","Data":"b94bd75cfddaad024fefa16f69de49bf7d33c358a3a1907a075d9928c84e808b"}
Oct 30 01:02:38.536159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:38.536008    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b94bd75cfddaad024fefa16f69de49bf7d33c358a3a1907a075d9928c84e808b"
Oct 30 01:02:38.783618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:38.783574    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:02:38.791210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:38.791128    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:02:38.792448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:38.792423    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:02:38.800673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:38.800651    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:02:39.158446 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:39.158314    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:02:40.558494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.558388    5104 generic.go:356] "Generic (PLEG): container finished" podID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerID="961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba" exitCode=0
Oct 30 01:02:40.558494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.558471    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" event={"ID":"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6","Type":"ContainerDied","Data":"961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba"}
Oct 30 01:02:40.558973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.558950    5104 scope.go:122] "RemoveContainer" containerID="961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba"
Oct 30 01:02:40.560688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.560661    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:40.560769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.560735    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:40.560814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.560769    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:02:40.945582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.924191    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"]
Oct 30 01:02:40.945582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.925424    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9df6a07d-d841-42ef-883a-7868e6db2c06" containerName="revoke-keys"
Oct 30 01:02:40.945582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.925445    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df6a07d-d841-42ef-883a-7868e6db2c06" containerName="revoke-keys"
Oct 30 01:02:40.945582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.925699    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df6a07d-d841-42ef-883a-7868e6db2c06" containerName="revoke-keys"
Oct 30 01:02:40.945582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.940898    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:40.962094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.956011    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:40.988939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.985624    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:40.988939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.985838    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:40.988939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:40.987443    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:02:41.000761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.000725    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"]
Oct 30 01:02:41.007156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.007117    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.007312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.007249    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.007450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.007431    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct4mw\" (UniqueName: \"kubernetes.io/projected/d50727b1-c604-43c2-a153-8bca30d2a775-kube-api-access-ct4mw\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.109400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.109351    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.109648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.109623    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct4mw\" (UniqueName: \"kubernetes.io/projected/d50727b1-c604-43c2-a153-8bca30d2a775-kube-api-access-ct4mw\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.109796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.109771    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.110488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.110445    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.112443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.112414    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.130076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.130050    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct4mw\" (UniqueName: \"kubernetes.io/projected/d50727b1-c604-43c2-a153-8bca30d2a775-kube-api-access-ct4mw\") pod \"maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.320013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.319917    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:41.571290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.571209    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" event={"ID":"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6","Type":"ContainerStarted","Data":"42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd"}
Oct 30 01:02:41.654642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.654609    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:02:41.654825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.654803    5104 patch_prober.go:29] interesting pod/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:02:41.654911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.654858    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:02:41.657198 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:02:41.657169    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50727b1_c604_43c2_a153_8bca30d2a775.slice/crio-172e1472d96dec65b13ed2a2bba94f80ef11522623413604642a5de6c9224918 WatchSource:0}: Error finding container 172e1472d96dec65b13ed2a2bba94f80ef11522623413604642a5de6c9224918: Status 404 returned error can't find the container with id 172e1472d96dec65b13ed2a2bba94f80ef11522623413604642a5de6c9224918
Oct 30 01:02:41.658612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:41.658591    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"]
Oct 30 01:02:42.614022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:42.613877    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f" event={"ID":"d50727b1-c604-43c2-a153-8bca30d2a775","Type":"ContainerStarted","Data":"006e2382e359f0ba5aa7d9f82b8c34e787ec2dc983106f197174f272cbea0ba1"}
Oct 30 01:02:42.614022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:42.613936    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f" event={"ID":"d50727b1-c604-43c2-a153-8bca30d2a775","Type":"ContainerStarted","Data":"172e1472d96dec65b13ed2a2bba94f80ef11522623413604642a5de6c9224918"}
Oct 30 01:02:42.654591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:42.654554    5104 patch_prober.go:29] interesting pod/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:02:42.654770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:42.654616    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:02:42.656101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:42.656055    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f" podStartSLOduration=2.6560409050000002 podStartE2EDuration="2.656040905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:42.649000322 +0000 UTC m=+1204.196690815" watchObservedRunningTime="2026-10-30 01:02:42.656040905 +0000 UTC m=+1204.203731431"
Oct 30 01:02:43.217885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:43.217841    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"]
Oct 30 01:02:43.251749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:43.251716    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"]
Oct 30 01:02:43.660099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:43.659993    5104 patch_prober.go:29] interesting pod/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:02:43.671659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:43.667573    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:02:43.686476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:43.682946    5104 generic.go:356] "Generic (PLEG): container finished" podID="d50727b1-c604-43c2-a153-8bca30d2a775" containerID="006e2382e359f0ba5aa7d9f82b8c34e787ec2dc983106f197174f272cbea0ba1" exitCode=0
Oct 30 01:02:43.686476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:43.683228    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f" event={"ID":"d50727b1-c604-43c2-a153-8bca30d2a775","Type":"ContainerDied","Data":"006e2382e359f0ba5aa7d9f82b8c34e787ec2dc983106f197174f272cbea0ba1"}
Oct 30 01:02:44.680494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.679623    5104 patch_prober.go:29] interesting pod/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:02:44.680494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.679690    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:02:44.826354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.824907    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"]
Oct 30 01:02:44.832721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.831664    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"]
Oct 30 01:02:44.832721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.831781    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:44.949212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.949111    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:44.949392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.949259    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ct9l\" (UniqueName: \"kubernetes.io/projected/706fa9b7-1056-43d9-9525-a5eb045c0e1d-kube-api-access-8ct9l\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:44.949392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:44.949305    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.051370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.051328    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.051585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.051417    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ct9l\" (UniqueName: \"kubernetes.io/projected/706fa9b7-1056-43d9-9525-a5eb045c0e1d-kube-api-access-8ct9l\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.051585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.051450    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.052539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.052511    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.058013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.057929    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.061349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.061318    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"]
Oct 30 01:02:45.109611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.109538    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"]
Oct 30 01:02:45.125861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.113440    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ct9l\" (UniqueName: \"kubernetes.io/projected/706fa9b7-1056-43d9-9525-a5eb045c0e1d-kube-api-access-8ct9l\") pod \"maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.140061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.133522    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc43d717-mldtb"]
Oct 30 01:02:45.146476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.146432    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08eba6-hxzdj"]
Oct 30 01:02:45.166472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.166432    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:45.220065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.219991    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:45.356656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.356626    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50727b1-c604-43c2-a153-8bca30d2a775-kube-api-access-ct4mw\" (UniqueName: \"kubernetes.io/projected/d50727b1-c604-43c2-a153-8bca30d2a775-kube-api-access-ct4mw\") pod \"d50727b1-c604-43c2-a153-8bca30d2a775\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") "
Oct 30 01:02:45.356833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.356796    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-openshift-ca\") pod \"d50727b1-c604-43c2-a153-8bca30d2a775\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") "
Oct 30 01:02:45.356833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.356821    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-xks-ca\") pod \"d50727b1-c604-43c2-a153-8bca30d2a775\" (UID: \"d50727b1-c604-43c2-a153-8bca30d2a775\") "
Oct 30 01:02:45.357874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.357841    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-openshift-ca" pod "d50727b1-c604-43c2-a153-8bca30d2a775" (UID: "d50727b1-c604-43c2-a153-8bca30d2a775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:45.360493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.360121    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-xks-ca" pod "d50727b1-c604-43c2-a153-8bca30d2a775" (UID: "d50727b1-c604-43c2-a153-8bca30d2a775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:45.360958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.360869    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50727b1-c604-43c2-a153-8bca30d2a775-kube-api-access-ct4mw" pod "d50727b1-c604-43c2-a153-8bca30d2a775" (UID: "d50727b1-c604-43c2-a153-8bca30d2a775"). InnerVolumeSpecName "kube-api-access-ct4mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:45.459840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.459801    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct4mw\" (UniqueName: \"kubernetes.io/projected/d50727b1-c604-43c2-a153-8bca30d2a775-kube-api-access-ct4mw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:45.459840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.459841    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:45.460149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.459857    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50727b1-c604-43c2-a153-8bca30d2a775-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:45.552847 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:02:45.552816    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706fa9b7_1056_43d9_9525_a5eb045c0e1d.slice/crio-d3ce417e07d13187e756306ec03c73502460d27fa4379c460e26cc8824e61055 WatchSource:0}: Error finding container d3ce417e07d13187e756306ec03c73502460d27fa4379c460e26cc8824e61055: Status 404 returned error can't find the container with id d3ce417e07d13187e756306ec03c73502460d27fa4379c460e26cc8824e61055
Oct 30 01:02:45.554403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.554380    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"]
Oct 30 01:02:45.654017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.653986    5104 patch_prober.go:29] interesting pod/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:02:45.654147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.654044    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:02:45.844863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.844829    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"
Oct 30 01:02:45.844863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.844843    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f" event={"ID":"d50727b1-c604-43c2-a153-8bca30d2a775","Type":"ContainerDied","Data":"172e1472d96dec65b13ed2a2bba94f80ef11522623413604642a5de6c9224918"}
Oct 30 01:02:45.845296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.844897    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="172e1472d96dec65b13ed2a2bba94f80ef11522623413604642a5de6c9224918"
Oct 30 01:02:45.846514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:45.846481    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w" event={"ID":"706fa9b7-1056-43d9-9525-a5eb045c0e1d","Type":"ContainerStarted","Data":"d3ce417e07d13187e756306ec03c73502460d27fa4379c460e26cc8824e61055"}
Oct 30 01:02:46.031651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.031615    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"]
Oct 30 01:02:46.043961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.043916    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3971a38-f57vn"]
Oct 30 01:02:46.612155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.612117    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a052ef3-84a6-442f-be2d-096077d985d5" path="/var/lib/kubelet/pods/4a052ef3-84a6-442f-be2d-096077d985d5/volumes"
Oct 30 01:02:46.612651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.612629    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d16483-dc6b-4ff6-bc16-a6c0606e6b23" path="/var/lib/kubelet/pods/a6d16483-dc6b-4ff6-bc16-a6c0606e6b23/volumes"
Oct 30 01:02:46.613099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.613083    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490" path="/var/lib/kubelet/pods/f56bf6d6-7d8c-4b66-90fe-6a16d4aa5490/volumes"
Oct 30 01:02:46.654890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.654859    5104 patch_prober.go:29] interesting pod/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:02:46.655065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.654909    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:02:46.859435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.859404    5104 generic.go:356] "Generic (PLEG): container finished" podID="706fa9b7-1056-43d9-9525-a5eb045c0e1d" containerID="cf4c5e99e690afc3aca7139854f8fe8f1df9c79f33b55c17e9c550a5173e2f0e" exitCode=0
Oct 30 01:02:46.859848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:46.859500    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w" event={"ID":"706fa9b7-1056-43d9-9525-a5eb045c0e1d","Type":"ContainerDied","Data":"cf4c5e99e690afc3aca7139854f8fe8f1df9c79f33b55c17e9c550a5173e2f0e"}
Oct 30 01:02:47.658316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:47.658288    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:02:47.658814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:47.658795    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:02:47.659431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:47.659414    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:02:48.231539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.224537    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw" podUID="8ae50783-4723-47c1-ba49-e85fd84a7321" containerName="payload-processing" containerID="cri-o://064e914c837bb8ceb53f4c68b0fe88ed16f495e2457123a96ca376145a74afeb" gracePeriod=25
Oct 30 01:02:48.254125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.253028    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt" podUID="21c23ee9-4928-4167-94c3-8c56160b186d" containerName="payload-pre-processing" containerID="cri-o://11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0" gracePeriod=25
Oct 30 01:02:48.757799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.755956    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"]
Oct 30 01:02:48.772758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.772657    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:48.813620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.813585    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-openshift-ca\") pod \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") "
Oct 30 01:02:48.813889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.813865    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706fa9b7-1056-43d9-9525-a5eb045c0e1d-kube-api-access-8ct9l\" (UniqueName: \"kubernetes.io/projected/706fa9b7-1056-43d9-9525-a5eb045c0e1d-kube-api-access-8ct9l\") pod \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") "
Oct 30 01:02:48.814167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.814150    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-xks-ca\") pod \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\" (UID: \"706fa9b7-1056-43d9-9525-a5eb045c0e1d\") "
Oct 30 01:02:48.863641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.861195    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-openshift-ca" pod "706fa9b7-1056-43d9-9525-a5eb045c0e1d" (UID: "706fa9b7-1056-43d9-9525-a5eb045c0e1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:48.928072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.917374    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:48.994491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.992580    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w" event={"ID":"706fa9b7-1056-43d9-9525-a5eb045c0e1d","Type":"ContainerDied","Data":"d3ce417e07d13187e756306ec03c73502460d27fa4379c460e26cc8824e61055"}
Oct 30 01:02:48.994772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:48.994748    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3ce417e07d13187e756306ec03c73502460d27fa4379c460e26cc8824e61055"
Oct 30 01:02:49.004873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.004382    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"
Oct 30 01:02:49.033051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.033017    5104 generic.go:356] "Generic (PLEG): container finished" podID="8ae50783-4723-47c1-ba49-e85fd84a7321" containerID="064e914c837bb8ceb53f4c68b0fe88ed16f495e2457123a96ca376145a74afeb" exitCode=0
Oct 30 01:02:49.042928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.042835    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw" event={"ID":"8ae50783-4723-47c1-ba49-e85fd84a7321","Type":"ContainerDied","Data":"064e914c837bb8ceb53f4c68b0fe88ed16f495e2457123a96ca376145a74afeb"}
Oct 30 01:02:49.043913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.043883    5104 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" secret="" err="secret \"e2e-ait-0660f91f-openshift-default-dockercfg-wl72z\" not found"
Oct 30 01:02:49.098816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.098778    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706fa9b7-1056-43d9-9525-a5eb045c0e1d-kube-api-access-8ct9l" pod "706fa9b7-1056-43d9-9525-a5eb045c0e1d" (UID: "706fa9b7-1056-43d9-9525-a5eb045c0e1d"). InnerVolumeSpecName "kube-api-access-8ct9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:49.124508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.122031    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-xks-ca" pod "706fa9b7-1056-43d9-9525-a5eb045c0e1d" (UID: "706fa9b7-1056-43d9-9525-a5eb045c0e1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:49.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.152376    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ct9l\" (UniqueName: \"kubernetes.io/projected/706fa9b7-1056-43d9-9525-a5eb045c0e1d-kube-api-access-8ct9l\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:49.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.152412    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706fa9b7-1056-43d9-9525-a5eb045c0e1d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:49.199545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.168985    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0660f91f-q49dq"]
Oct 30 01:02:49.294821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.294787    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"]
Oct 30 01:02:49.721670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.721644    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:02:49.746361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.746332    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae50783-4723-47c1-ba49-e85fd84a7321-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae50783-4723-47c1-ba49-e85fd84a7321-plugins-config-volume\") pod \"8ae50783-4723-47c1-ba49-e85fd84a7321\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") "
Oct 30 01:02:49.746613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.746597    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae50783-4723-47c1-ba49-e85fd84a7321-kube-api-access-79jgh\" (UniqueName: \"kubernetes.io/projected/8ae50783-4723-47c1-ba49-e85fd84a7321-kube-api-access-79jgh\") pod \"8ae50783-4723-47c1-ba49-e85fd84a7321\" (UID: \"8ae50783-4723-47c1-ba49-e85fd84a7321\") "
Oct 30 01:02:49.746747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.746724    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae50783-4723-47c1-ba49-e85fd84a7321-plugins-config-volume" pod "8ae50783-4723-47c1-ba49-e85fd84a7321" (UID: "8ae50783-4723-47c1-ba49-e85fd84a7321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:49.747085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.747065    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae50783-4723-47c1-ba49-e85fd84a7321-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:49.748723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.748701    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae50783-4723-47c1-ba49-e85fd84a7321-kube-api-access-79jgh" pod "8ae50783-4723-47c1-ba49-e85fd84a7321" (UID: "8ae50783-4723-47c1-ba49-e85fd84a7321"). InnerVolumeSpecName "kube-api-access-79jgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:49.770388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.770263    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:02:49.848795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.848690    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c23ee9-4928-4167-94c3-8c56160b186d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c23ee9-4928-4167-94c3-8c56160b186d-plugins-config-volume\") pod \"21c23ee9-4928-4167-94c3-8c56160b186d\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") "
Oct 30 01:02:49.848981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.848959    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c23ee9-4928-4167-94c3-8c56160b186d-kube-api-access-k2vtw\" (UniqueName: \"kubernetes.io/projected/21c23ee9-4928-4167-94c3-8c56160b186d-kube-api-access-k2vtw\") pod \"21c23ee9-4928-4167-94c3-8c56160b186d\" (UID: \"21c23ee9-4928-4167-94c3-8c56160b186d\") "
Oct 30 01:02:49.849512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.849434    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c23ee9-4928-4167-94c3-8c56160b186d-plugins-config-volume" pod "21c23ee9-4928-4167-94c3-8c56160b186d" (UID: "21c23ee9-4928-4167-94c3-8c56160b186d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:49.849853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.849831    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79jgh\" (UniqueName: \"kubernetes.io/projected/8ae50783-4723-47c1-ba49-e85fd84a7321-kube-api-access-79jgh\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:49.849925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.849859    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c23ee9-4928-4167-94c3-8c56160b186d-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:49.851894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.851869    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c23ee9-4928-4167-94c3-8c56160b186d-kube-api-access-k2vtw" pod "21c23ee9-4928-4167-94c3-8c56160b186d" (UID: "21c23ee9-4928-4167-94c3-8c56160b186d"). InnerVolumeSpecName "kube-api-access-k2vtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:49.951816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:49.951769    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vtw\" (UniqueName: \"kubernetes.io/projected/21c23ee9-4928-4167-94c3-8c56160b186d-kube-api-access-k2vtw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:50.172541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.169854    5104 generic.go:356] "Generic (PLEG): container finished" podID="21c23ee9-4928-4167-94c3-8c56160b186d" containerID="11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0" exitCode=0
Oct 30 01:02:50.172541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.170221    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt" event={"ID":"21c23ee9-4928-4167-94c3-8c56160b186d","Type":"ContainerDied","Data":"11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0"}
Oct 30 01:02:50.172541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.170298    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt" event={"ID":"21c23ee9-4928-4167-94c3-8c56160b186d","Type":"ContainerDied","Data":"545551ce784181d523f2596160e16e1c165b4e734ad2c46a1bcc83934477309d"}
Oct 30 01:02:50.172541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.170333    5104 scope.go:122] "RemoveContainer" containerID="11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0"
Oct 30 01:02:50.184684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.184653    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"
Oct 30 01:02:50.285889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.285820    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy" containerID="cri-o://42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd" gracePeriod=30
Oct 30 01:02:50.286660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.286630    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"
Oct 30 01:02:50.310480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.292554    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw" event={"ID":"8ae50783-4723-47c1-ba49-e85fd84a7321","Type":"ContainerDied","Data":"020edfb6ded6b9ec70b5ac3ed6bf04609be54faed8aa608f9075d64448a2beba"}
Oct 30 01:02:50.348749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.348719    5104 scope.go:122] "RemoveContainer" containerID="11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0"
Oct 30 01:02:50.350868 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:50.350839    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0\": container with ID starting with 11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0 not found: ID does not exist" containerID="11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0"
Oct 30 01:02:50.351083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.351055    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0"} err="failed to get container status \"11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0\": rpc error: code = NotFound desc = could not find container \"11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0\": container with ID starting with 11c143980b5dcf1561b73d40db4b99442caf5d47f5d9cae27412faaf55d172d0 not found: ID does not exist"
Oct 30 01:02:50.351213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.351200    5104 scope.go:122] "RemoveContainer" containerID="064e914c837bb8ceb53f4c68b0fe88ed16f495e2457123a96ca376145a74afeb"
Oct 30 01:02:50.634851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.634162    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df6a07d-d841-42ef-883a-7868e6db2c06" path="/var/lib/kubelet/pods/9df6a07d-d841-42ef-883a-7868e6db2c06/volumes"
Oct 30 01:02:50.667549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.667519    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"]
Oct 30 01:02:50.670869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.670842    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0660f91f-6f55c5558c-gbhqt"]
Oct 30 01:02:50.708122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.704742    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"]
Oct 30 01:02:50.713822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:50.713781    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0660f91f-c4956954b-s2lkw"]
Oct 30 01:02:51.337739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:51.337699    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"]
Oct 30 01:02:51.369332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:51.369300    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"]
Oct 30 01:02:52.788589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:52.788545    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c23ee9-4928-4167-94c3-8c56160b186d" path="/var/lib/kubelet/pods/21c23ee9-4928-4167-94c3-8c56160b186d/volumes"
Oct 30 01:02:52.789224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:52.789101    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae50783-4723-47c1-ba49-e85fd84a7321" path="/var/lib/kubelet/pods/8ae50783-4723-47c1-ba49-e85fd84a7321/volumes"
Oct 30 01:02:52.956309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:52.956269    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:02:53.460901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:53.460867    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"]
Oct 30 01:02:53.481026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:53.480877    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8b58192f-l8n8f"]
Oct 30 01:02:53.603521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:53.603491    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"]
Oct 30 01:02:53.604005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:53.603968    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78" podUID="52d47ab9-6401-40f8-9533-756429e691f9" containerName="istio-proxy" containerID="cri-o://ed1a4debad22adba3bf694a476ef9d3ff38780964e624ca0263ec5a2182962bb" gracePeriod=30
Oct 30 01:02:54.579793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:54.579754    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50727b1-c604-43c2-a153-8bca30d2a775" path="/var/lib/kubelet/pods/d50727b1-c604-43c2-a153-8bca30d2a775/volumes"
Oct 30 01:02:55.706681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:55.706645    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"]
Oct 30 01:02:55.758092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:55.757723    5104 patch_prober.go:29] interesting pod/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:02:55.758092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:55.757791    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78" podUID="52d47ab9-6401-40f8-9533-756429e691f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:02:56.117816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.117780    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"]
Oct 30 01:02:56.139273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.139237    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18f3a6a-dv2pj"]
Oct 30 01:02:56.145280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.145251    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:02:56.222818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.222781    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-credential-socket\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.222818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.222825    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-token\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-token\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.222947    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istiod-ca-cert\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223120    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-kube-api-access-pmkcc\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-kube-api-access-pmkcc\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223178    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-envoy\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223207    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-socket\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223285    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-data\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223326    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-certs\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223423    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-podinfo\") pod \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\" (UID: \"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6\") "
Oct 30 01:02:56.223529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223118    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-credential-socket" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:56.223600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223415    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istiod-ca-cert" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:56.223764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223732    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-socket" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:56.224087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.223839    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-certs" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:56.224308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.224283    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.224447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.224391    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.224447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.224412    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.224447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.224414    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-data" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:56.224447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.224429    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.226068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.226031    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-kube-api-access-pmkcc" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "kube-api-access-pmkcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:56.226259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.226232    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-token" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:56.226435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.226418    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-envoy" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:56.226569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.226498    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-podinfo" pod "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" (UID: "3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:02:56.326138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.326097    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkcc\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-kube-api-access-pmkcc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.326138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.326132    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.326138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.326141    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.326138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.326149    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.326421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.326157    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:56.338540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.338482    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf" podUID="027a45ec-0dc6-4960-ae70-f66a54134342" containerName="payload-processing" containerID="cri-o://b01a4334a66d49b920dde98ad3b24fc401f317554658d29b00ca5767c641a181" gracePeriod=25
Oct 30 01:02:56.370287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.370165    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66" podUID="f8fc1b38-32cc-4853-824e-f3938c66e86e" containerName="payload-pre-processing" containerID="cri-o://f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f" gracePeriod=25
Oct 30 01:02:56.415192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.415161    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"]
Oct 30 01:02:56.415992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.415976    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d50727b1-c604-43c2-a153-8bca30d2a775" containerName="revoke-keys"
Oct 30 01:02:56.416048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.415995    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50727b1-c604-43c2-a153-8bca30d2a775" containerName="revoke-keys"
Oct 30 01:02:56.416048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416011    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy"
Oct 30 01:02:56.416048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416016    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy"
Oct 30 01:02:56.416048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416038    5104 cpu_manager.go:414] "containerMap: removing container" podUID="21c23ee9-4928-4167-94c3-8c56160b186d" containerName="payload-pre-processing"
Oct 30 01:02:56.416048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416044    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c23ee9-4928-4167-94c3-8c56160b186d" containerName="payload-pre-processing"
Oct 30 01:02:56.416207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416056    5104 cpu_manager.go:414] "containerMap: removing container" podUID="706fa9b7-1056-43d9-9525-a5eb045c0e1d" containerName="revoke-keys"
Oct 30 01:02:56.416207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416060    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706fa9b7-1056-43d9-9525-a5eb045c0e1d" containerName="revoke-keys"
Oct 30 01:02:56.416207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416068    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8ae50783-4723-47c1-ba49-e85fd84a7321" containerName="payload-processing"
Oct 30 01:02:56.416207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416073    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae50783-4723-47c1-ba49-e85fd84a7321" containerName="payload-processing"
Oct 30 01:02:56.416207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416187    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50727b1-c604-43c2-a153-8bca30d2a775" containerName="revoke-keys"
Oct 30 01:02:56.416207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416200    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy"
Oct 30 01:02:56.416377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416242    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c23ee9-4928-4167-94c3-8c56160b186d" containerName="payload-pre-processing"
Oct 30 01:02:56.416377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416253    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="706fa9b7-1056-43d9-9525-a5eb045c0e1d" containerName="revoke-keys"
Oct 30 01:02:56.416377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.416265    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae50783-4723-47c1-ba49-e85fd84a7321" containerName="payload-processing"
Oct 30 01:02:56.420425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.420403    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.442518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.436566    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:56.442518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.436630    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:56.442518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.439786    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"]
Oct 30 01:02:56.518477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.505939    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:02:56.535500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.527573    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:56.556396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.556357    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.556710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.556682    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.556940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.556923    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4db\" (UniqueName: \"kubernetes.io/projected/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-kube-api-access-nn4db\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.675529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.670486    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.675529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.670536    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.675529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.670664    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4db\" (UniqueName: \"kubernetes.io/projected/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-kube-api-access-nn4db\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.675529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.672121    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.677309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.677268    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce60ea83-54d1-4103-9d18-e3ed4b201ec7" path="/var/lib/kubelet/pods/ce60ea83-54d1-4103-9d18-e3ed4b201ec7/volumes"
Oct 30 01:02:56.875564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.803039    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4db\" (UniqueName: \"kubernetes.io/projected/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-kube-api-access-nn4db\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.892479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.877048    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d283a11e-59spf\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.910529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.903832    5104 generic.go:356] "Generic (PLEG): container finished" podID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerID="42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd" exitCode=0
Oct 30 01:02:56.910529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.904214    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" event={"ID":"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6","Type":"ContainerDied","Data":"42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd"}
Oct 30 01:02:56.910529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.904278    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz" event={"ID":"3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6","Type":"ContainerDied","Data":"2da29f0d080b51b446acba390496cdc52b0c8f26a92de37d26d27af974f89bd6"}
Oct 30 01:02:56.910529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.904311    5104 scope.go:122] "RemoveContainer" containerID="42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd"
Oct 30 01:02:56.910529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.904347    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"
Oct 30 01:02:56.925064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.924753    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:02:56.929732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.929434    5104 generic.go:356] "Generic (PLEG): container finished" podID="027a45ec-0dc6-4960-ae70-f66a54134342" containerID="b01a4334a66d49b920dde98ad3b24fc401f317554658d29b00ca5767c641a181" exitCode=0
Oct 30 01:02:56.929858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.929776    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf" event={"ID":"027a45ec-0dc6-4960-ae70-f66a54134342","Type":"ContainerDied","Data":"b01a4334a66d49b920dde98ad3b24fc401f317554658d29b00ca5767c641a181"}
Oct 30 01:02:56.969875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.969783    5104 scope.go:122] "RemoveContainer" containerID="961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba"
Oct 30 01:02:56.987788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.987755    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"]
Oct 30 01:02:56.998090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:56.997945    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0660f91f-openshift-default-5b87f4869c-56djz"]
Oct 30 01:02:57.014885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.014859    5104 scope.go:122] "RemoveContainer" containerID="42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd"
Oct 30 01:02:57.015351 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:57.015312    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd\": container with ID starting with 42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd not found: ID does not exist" containerID="42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd"
Oct 30 01:02:57.015485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.015356    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd"} err="failed to get container status \"42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd\": rpc error: code = NotFound desc = could not find container \"42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd\": container with ID starting with 42530e5f1984c81e62181f99462cd06e5917d26122866455f3c23446752c75dd not found: ID does not exist"
Oct 30 01:02:57.015485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.015382    5104 scope.go:122] "RemoveContainer" containerID="961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba"
Oct 30 01:02:57.015997 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:57.015973    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba\": container with ID starting with 961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba not found: ID does not exist" containerID="961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba"
Oct 30 01:02:57.016067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.016006    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba"} err="failed to get container status \"961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba\": rpc error: code = NotFound desc = could not find container \"961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba\": container with ID starting with 961cc328aa1c461e79ca3a072f3c6d86bd2b998c9b802a06a8451cf827c10dba not found: ID does not exist"
Oct 30 01:02:57.130064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.129985    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:57.194415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.194388    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:02:57.208057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.208024    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027a45ec-0dc6-4960-ae70-f66a54134342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027a45ec-0dc6-4960-ae70-f66a54134342-plugins-config-volume\") pod \"027a45ec-0dc6-4960-ae70-f66a54134342\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") "
Oct 30 01:02:57.208225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.208082    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027a45ec-0dc6-4960-ae70-f66a54134342-kube-api-access-mk7s9\" (UniqueName: \"kubernetes.io/projected/027a45ec-0dc6-4960-ae70-f66a54134342-kube-api-access-mk7s9\") pod \"027a45ec-0dc6-4960-ae70-f66a54134342\" (UID: \"027a45ec-0dc6-4960-ae70-f66a54134342\") "
Oct 30 01:02:57.208500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.208472    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027a45ec-0dc6-4960-ae70-f66a54134342-plugins-config-volume" pod "027a45ec-0dc6-4960-ae70-f66a54134342" (UID: "027a45ec-0dc6-4960-ae70-f66a54134342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:57.208736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.208717    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027a45ec-0dc6-4960-ae70-f66a54134342-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:57.210364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.210341    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027a45ec-0dc6-4960-ae70-f66a54134342-kube-api-access-mk7s9" pod "027a45ec-0dc6-4960-ae70-f66a54134342" (UID: "027a45ec-0dc6-4960-ae70-f66a54134342"). InnerVolumeSpecName "kube-api-access-mk7s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:57.304184 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:02:57.304150    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50167c9_f5fe_45b9_83b1_8f495bfc65c0.slice/crio-5e19db62b5e75da345474ca5f3a8b1ae33615cac4022e24d81406a0d3ea4d8cd WatchSource:0}: Error finding container 5e19db62b5e75da345474ca5f3a8b1ae33615cac4022e24d81406a0d3ea4d8cd: Status 404 returned error can't find the container with id 5e19db62b5e75da345474ca5f3a8b1ae33615cac4022e24d81406a0d3ea4d8cd
Oct 30 01:02:57.305681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.305658    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"]
Oct 30 01:02:57.309727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.309285    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8fc1b38-32cc-4853-824e-f3938c66e86e-kube-api-access-2jscx\" (UniqueName: \"kubernetes.io/projected/f8fc1b38-32cc-4853-824e-f3938c66e86e-kube-api-access-2jscx\") pod \"f8fc1b38-32cc-4853-824e-f3938c66e86e\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") "
Oct 30 01:02:57.309727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.309397    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8fc1b38-32cc-4853-824e-f3938c66e86e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc1b38-32cc-4853-824e-f3938c66e86e-plugins-config-volume\") pod \"f8fc1b38-32cc-4853-824e-f3938c66e86e\" (UID: \"f8fc1b38-32cc-4853-824e-f3938c66e86e\") "
Oct 30 01:02:57.309861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.309819    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fc1b38-32cc-4853-824e-f3938c66e86e-plugins-config-volume" pod "f8fc1b38-32cc-4853-824e-f3938c66e86e" (UID: "f8fc1b38-32cc-4853-824e-f3938c66e86e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:57.310051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.310032    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7s9\" (UniqueName: \"kubernetes.io/projected/027a45ec-0dc6-4960-ae70-f66a54134342-kube-api-access-mk7s9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:57.310087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.310057    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc1b38-32cc-4853-824e-f3938c66e86e-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:57.311868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.311846    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fc1b38-32cc-4853-824e-f3938c66e86e-kube-api-access-2jscx" pod "f8fc1b38-32cc-4853-824e-f3938c66e86e" (UID: "f8fc1b38-32cc-4853-824e-f3938c66e86e"). InnerVolumeSpecName "kube-api-access-2jscx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:57.412253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.412216    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jscx\" (UniqueName: \"kubernetes.io/projected/f8fc1b38-32cc-4853-824e-f3938c66e86e-kube-api-access-2jscx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:57.950023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.947978    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf" event={"ID":"a50167c9-f5fe-45b9-83b1-8f495bfc65c0","Type":"ContainerStarted","Data":"b3b9230f2548e8a950e0fc08c296b0a9a54cf310d1826aa734df796d8fd683e6"}
Oct 30 01:02:57.950023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.948045    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf" event={"ID":"a50167c9-f5fe-45b9-83b1-8f495bfc65c0","Type":"ContainerStarted","Data":"5e19db62b5e75da345474ca5f3a8b1ae33615cac4022e24d81406a0d3ea4d8cd"}
Oct 30 01:02:57.956516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.952607    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf" event={"ID":"027a45ec-0dc6-4960-ae70-f66a54134342","Type":"ContainerDied","Data":"d791de638dce696ad554cac8bf0afb5ca67290bfb513add63592c775628fa6ae"}
Oct 30 01:02:57.956516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.952666    5104 scope.go:122] "RemoveContainer" containerID="b01a4334a66d49b920dde98ad3b24fc401f317554658d29b00ca5767c641a181"
Oct 30 01:02:57.956516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.952781    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"
Oct 30 01:02:57.961964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.960051    5104 generic.go:356] "Generic (PLEG): container finished" podID="f8fc1b38-32cc-4853-824e-f3938c66e86e" containerID="f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f" exitCode=0
Oct 30 01:02:57.961964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.960321    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"
Oct 30 01:02:57.961964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.961293    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66" event={"ID":"f8fc1b38-32cc-4853-824e-f3938c66e86e","Type":"ContainerDied","Data":"f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f"}
Oct 30 01:02:57.961964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.961344    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66" event={"ID":"f8fc1b38-32cc-4853-824e-f3938c66e86e","Type":"ContainerDied","Data":"5eb8ea3c727a9879209b76495a736f51ece1e9d537912ec449c4c17a9cddd465"}
Oct 30 01:02:57.975741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.973629    5104 scope.go:122] "RemoveContainer" containerID="f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f"
Oct 30 01:02:57.989536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.989515    5104 scope.go:122] "RemoveContainer" containerID="f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f"
Oct 30 01:02:57.989890 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:02:57.989863    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f\": container with ID starting with f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f not found: ID does not exist" containerID="f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f"
Oct 30 01:02:57.990031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.989900    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f"} err="failed to get container status \"f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f\": rpc error: code = NotFound desc = could not find container \"f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f\": container with ID starting with f4852c7e0eca2e5bbf378c14c6653be4909871f29944926150a00c80136ec87f not found: ID does not exist"
Oct 30 01:02:57.991187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:57.991141    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf" podStartSLOduration=1.991128975 podStartE2EDuration="1.991128975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:57.986547764 +0000 UTC m=+1219.534238260" watchObservedRunningTime="2026-10-30 01:02:57.991128975 +0000 UTC m=+1219.538819471"
Oct 30 01:02:58.026742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.026716    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"]
Oct 30 01:02:58.033921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.033332    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"]
Oct 30 01:02:58.038836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.038812    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8b58192f-65f9c8cdf9-tqh66"]
Oct 30 01:02:58.043668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.043645    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbc9862-jmw4n"]
Oct 30 01:02:58.063850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.063823    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"]
Oct 30 01:02:58.074540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.073201    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8b58192f-f48df6874-wbtnf"]
Oct 30 01:02:58.731499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.722191    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027a45ec-0dc6-4960-ae70-f66a54134342" path="/var/lib/kubelet/pods/027a45ec-0dc6-4960-ae70-f66a54134342/volumes"
Oct 30 01:02:58.740569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.738826    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" path="/var/lib/kubelet/pods/3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6/volumes"
Oct 30 01:02:58.740569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.739899    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a9f0a2d-d142-4da1-83b4-e4571c3c3755" path="/var/lib/kubelet/pods/4a9f0a2d-d142-4da1-83b4-e4571c3c3755/volumes"
Oct 30 01:02:58.740846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.740759    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8fc1b38-32cc-4853-824e-f3938c66e86e" path="/var/lib/kubelet/pods/f8fc1b38-32cc-4853-824e-f3938c66e86e/volumes"
Oct 30 01:02:58.745045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.741932    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"]
Oct 30 01:02:58.745045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.744236    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b61ed672-sqt6w"]
Oct 30 01:02:58.985759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.985012    5104 generic.go:356] "Generic (PLEG): container finished" podID="52d47ab9-6401-40f8-9533-756429e691f9" containerID="ed1a4debad22adba3bf694a476ef9d3ff38780964e624ca0263ec5a2182962bb" exitCode=0
Oct 30 01:02:58.985759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.985195    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78" event={"ID":"52d47ab9-6401-40f8-9533-756429e691f9","Type":"ContainerDied","Data":"ed1a4debad22adba3bf694a476ef9d3ff38780964e624ca0263ec5a2182962bb"}
Oct 30 01:02:58.989189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.988937    5104 generic.go:356] "Generic (PLEG): container finished" podID="a50167c9-f5fe-45b9-83b1-8f495bfc65c0" containerID="b3b9230f2548e8a950e0fc08c296b0a9a54cf310d1826aa734df796d8fd683e6" exitCode=0
Oct 30 01:02:58.989189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:58.989154    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf" event={"ID":"a50167c9-f5fe-45b9-83b1-8f495bfc65c0","Type":"ContainerDied","Data":"b3b9230f2548e8a950e0fc08c296b0a9a54cf310d1826aa734df796d8fd683e6"}
Oct 30 01:02:59.065750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.065714    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:02:59.205427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205323    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-istio-token\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-istio-token\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205403    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-certs\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205505    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52d47ab9-6401-40f8-9533-756429e691f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d47ab9-6401-40f8-9533-756429e691f9-istio-podinfo\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205611    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-credential-socket\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205645    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-kube-api-access-nvslf\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-kube-api-access-nvslf\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205737    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-certs" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:59.205873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205760    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-socket\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205810    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d47ab9-6401-40f8-9533-756429e691f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d47ab9-6401-40f8-9533-756429e691f9-istiod-ca-cert\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205848    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-envoy\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.205873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205870    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-credential-socket" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:59.206130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.205901    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-data\") pod \"52d47ab9-6401-40f8-9533-756429e691f9\" (UID: \"52d47ab9-6401-40f8-9533-756429e691f9\") "
Oct 30 01:02:59.206130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206038    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-socket" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:59.206242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206196    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d47ab9-6401-40f8-9533-756429e691f9-istiod-ca-cert" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:59.206337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206316    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-data" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:59.206807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206783    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.206940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206808    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.206940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206824    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d47ab9-6401-40f8-9533-756429e691f9-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.206940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206837    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.206940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.206850    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.208005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.207979    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-istio-token" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:59.208172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.208154    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-envoy" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:59.208360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.208347    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52d47ab9-6401-40f8-9533-756429e691f9-istio-podinfo" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:02:59.208524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.208508    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-kube-api-access-nvslf" pod "52d47ab9-6401-40f8-9533-756429e691f9" (UID: "52d47ab9-6401-40f8-9533-756429e691f9"). InnerVolumeSpecName "kube-api-access-nvslf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:59.309053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.309017    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d47ab9-6401-40f8-9533-756429e691f9-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.309053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.309056    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvslf\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-kube-api-access-nvslf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.309266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.309071    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d47ab9-6401-40f8-9533-756429e691f9-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.309266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:02:59.309087    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d47ab9-6401-40f8-9533-756429e691f9-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.007114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.007083    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"
Oct 30 01:03:00.007620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.007106    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78" event={"ID":"52d47ab9-6401-40f8-9533-756429e691f9","Type":"ContainerDied","Data":"60186ad79476c6ddee3805319a0bee35b0d6f677dfdf011c8aac44ecf306989c"}
Oct 30 01:03:00.007620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.007181    5104 scope.go:122] "RemoveContainer" containerID="ed1a4debad22adba3bf694a476ef9d3ff38780964e624ca0263ec5a2182962bb"
Oct 30 01:03:00.072221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.072186    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"]
Oct 30 01:03:00.107521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.081805    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8b58192f-openshift-default-754c75cb97-qdm78"]
Oct 30 01:03:00.213334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.213305    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:03:00.357532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.348634    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-openshift-ca\") pod \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") "
Oct 30 01:03:00.357532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.348795    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-kube-api-access-nn4db\" (UniqueName: \"kubernetes.io/projected/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-kube-api-access-nn4db\") pod \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") "
Oct 30 01:03:00.357532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.348888    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-xks-ca\") pod \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\" (UID: \"a50167c9-f5fe-45b9-83b1-8f495bfc65c0\") "
Oct 30 01:03:00.395481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.395367    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-openshift-ca" pod "a50167c9-f5fe-45b9-83b1-8f495bfc65c0" (UID: "a50167c9-f5fe-45b9-83b1-8f495bfc65c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:00.422480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.422208    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.537095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.524672    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-kube-api-access-nn4db" pod "a50167c9-f5fe-45b9-83b1-8f495bfc65c0" (UID: "a50167c9-f5fe-45b9-83b1-8f495bfc65c0"). InnerVolumeSpecName "kube-api-access-nn4db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:00.553500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.537500    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4db\" (UniqueName: \"kubernetes.io/projected/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-kube-api-access-nn4db\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.620687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.619876    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-xks-ca" pod "a50167c9-f5fe-45b9-83b1-8f495bfc65c0" (UID: "a50167c9-f5fe-45b9-83b1-8f495bfc65c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:00.676694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.675851    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50167c9-f5fe-45b9-83b1-8f495bfc65c0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.707726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.707679    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk" podUID="2f565644-a0ed-430d-bc62-e3bac1584495" containerName="payload-processing" containerID="cri-o://acd71c90717706703082ecb51a837a296a9272839d88c937513cf63711f766e2" gracePeriod=25
Oct 30 01:03:00.811490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.800341    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d47ab9-6401-40f8-9533-756429e691f9" path="/var/lib/kubelet/pods/52d47ab9-6401-40f8-9533-756429e691f9/volumes"
Oct 30 01:03:00.833055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:00.817019    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706fa9b7-1056-43d9-9525-a5eb045c0e1d" path="/var/lib/kubelet/pods/706fa9b7-1056-43d9-9525-a5eb045c0e1d/volumes"
Oct 30 01:03:01.022730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.022688    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf" event={"ID":"a50167c9-f5fe-45b9-83b1-8f495bfc65c0","Type":"ContainerDied","Data":"5e19db62b5e75da345474ca5f3a8b1ae33615cac4022e24d81406a0d3ea4d8cd"}
Oct 30 01:03:01.023229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.022754    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e19db62b5e75da345474ca5f3a8b1ae33615cac4022e24d81406a0d3ea4d8cd"
Oct 30 01:03:01.023229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.022873    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"
Oct 30 01:03:01.025964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.025933    5104 generic.go:356] "Generic (PLEG): container finished" podID="2f565644-a0ed-430d-bc62-e3bac1584495" containerID="acd71c90717706703082ecb51a837a296a9272839d88c937513cf63711f766e2" exitCode=0
Oct 30 01:03:01.026241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.026211    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk" event={"ID":"2f565644-a0ed-430d-bc62-e3bac1584495","Type":"ContainerDied","Data":"acd71c90717706703082ecb51a837a296a9272839d88c937513cf63711f766e2"}
Oct 30 01:03:01.060013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.059982    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:03:01.125529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.120112    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f565644-a0ed-430d-bc62-e3bac1584495-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f565644-a0ed-430d-bc62-e3bac1584495-plugins-config-volume\") pod \"2f565644-a0ed-430d-bc62-e3bac1584495\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") "
Oct 30 01:03:01.125529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.120451    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f565644-a0ed-430d-bc62-e3bac1584495-kube-api-access-z9h54\" (UniqueName: \"kubernetes.io/projected/2f565644-a0ed-430d-bc62-e3bac1584495-kube-api-access-z9h54\") pod \"2f565644-a0ed-430d-bc62-e3bac1584495\" (UID: \"2f565644-a0ed-430d-bc62-e3bac1584495\") "
Oct 30 01:03:01.125529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.124112    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f565644-a0ed-430d-bc62-e3bac1584495-plugins-config-volume" pod "2f565644-a0ed-430d-bc62-e3bac1584495" (UID: "2f565644-a0ed-430d-bc62-e3bac1584495"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:01.125529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.124220    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f565644-a0ed-430d-bc62-e3bac1584495-kube-api-access-z9h54" pod "2f565644-a0ed-430d-bc62-e3bac1584495" (UID: "2f565644-a0ed-430d-bc62-e3bac1584495"). InnerVolumeSpecName "kube-api-access-z9h54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:01.223181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.223137    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9h54\" (UniqueName: \"kubernetes.io/projected/2f565644-a0ed-430d-bc62-e3bac1584495-kube-api-access-z9h54\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.223181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:01.223173    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f565644-a0ed-430d-bc62-e3bac1584495-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:02.036655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:02.036627    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"
Oct 30 01:03:02.037136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:02.036628    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk" event={"ID":"2f565644-a0ed-430d-bc62-e3bac1584495","Type":"ContainerDied","Data":"351a2951a145a733ded57542f9ef71e160d86a673f7613ff83661871235596ca"}
Oct 30 01:03:02.037136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:02.036769    5104 scope.go:122] "RemoveContainer" containerID="acd71c90717706703082ecb51a837a296a9272839d88c937513cf63711f766e2"
Oct 30 01:03:02.089373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:02.089338    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"]
Oct 30 01:03:02.099705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:02.099673    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b61ed672-65b7c7796d-5g7zk"]
Oct 30 01:03:02.582661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:02.581828    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f565644-a0ed-430d-bc62-e3bac1584495" path="/var/lib/kubelet/pods/2f565644-a0ed-430d-bc62-e3bac1584495/volumes"
Oct 30 01:03:05.028261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:05.028226    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"]
Oct 30 01:03:05.040005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:05.039969    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"]
Oct 30 01:03:05.045699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:05.045666    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3ab9331-qrwmg"]
Oct 30 01:03:05.051013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:05.050988    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb340ac-nvrhf"]
Oct 30 01:03:06.027437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:06.027404    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"]
Oct 30 01:03:06.038635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:06.038608    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a25de51f-742cq"]
Oct 30 01:03:06.581736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:06.581699    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62cb1040-f3ab-4784-b08d-e327832ef901" path="/var/lib/kubelet/pods/62cb1040-f3ab-4784-b08d-e327832ef901/volumes"
Oct 30 01:03:06.582246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:06.582230    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a40d1e5-98a5-47a8-b1aa-5d5539709b34" path="/var/lib/kubelet/pods/9a40d1e5-98a5-47a8-b1aa-5d5539709b34/volumes"
Oct 30 01:03:06.582721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:06.582707    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1548dd-6b73-479c-9060-ee13609ade59" path="/var/lib/kubelet/pods/ff1548dd-6b73-479c-9060-ee13609ade59/volumes"
Oct 30 01:03:09.414486 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:03:09.413636    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:03:10.771495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:10.760635    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:03:12.028446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:12.028407    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"]
Oct 30 01:03:12.038540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:12.038502    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec74fc3-4bkfb"]
Oct 30 01:03:12.578938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:12.578898    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bdc66c-fdaf-4e19-be14-58b9cdacbc5b" path="/var/lib/kubelet/pods/07bdc66c-fdaf-4e19-be14-58b9cdacbc5b/volumes"
Oct 30 01:03:15.084373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:15.084330    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"]
Oct 30 01:03:15.095235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:15.095202    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858cc330-gxdct"]
Oct 30 01:03:16.579212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:16.579180    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfc690e-c9f9-4c5a-9518-6668feda534c" path="/var/lib/kubelet/pods/5bfc690e-c9f9-4c5a-9518-6668feda534c/volumes"
Oct 30 01:03:17.030401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:17.030359    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"]
Oct 30 01:03:17.041025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:17.040990    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97829e2-x5r6h"]
Oct 30 01:03:18.593116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:18.593074    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdf9d33-647e-460b-9186-5ea2c337078a" path="/var/lib/kubelet/pods/8fdf9d33-647e-460b-9186-5ea2c337078a/volumes"
Oct 30 01:03:19.915523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:19.914688    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:03:28.028678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.028639    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"]
Oct 30 01:03:28.047407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.047370    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aba9680f-qj6sh"]
Oct 30 01:03:28.578737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.578703    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2122562-9974-47ec-8143-039fdb5cb303" path="/var/lib/kubelet/pods/c2122562-9974-47ec-8143-039fdb5cb303/volumes"
Oct 30 01:03:28.814938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.814902    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"]
Oct 30 01:03:28.815724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815701    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f8fc1b38-32cc-4853-824e-f3938c66e86e" containerName="payload-pre-processing"
Oct 30 01:03:28.815724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815723    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fc1b38-32cc-4853-824e-f3938c66e86e" containerName="payload-pre-processing"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815755    5104 cpu_manager.go:414] "containerMap: removing container" podUID="52d47ab9-6401-40f8-9533-756429e691f9" containerName="istio-proxy"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815763    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d47ab9-6401-40f8-9533-756429e691f9" containerName="istio-proxy"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815804    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2f565644-a0ed-430d-bc62-e3bac1584495" containerName="payload-processing"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815812    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f565644-a0ed-430d-bc62-e3bac1584495" containerName="payload-processing"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815823    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815830    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815841    5104 cpu_manager.go:414] "containerMap: removing container" podUID="027a45ec-0dc6-4960-ae70-f66a54134342" containerName="payload-processing"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815848    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027a45ec-0dc6-4960-ae70-f66a54134342" containerName="payload-processing"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815878    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a50167c9-f5fe-45b9-83b1-8f495bfc65c0" containerName="revoke-keys"
Oct 30 01:03:28.815935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.815885    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50167c9-f5fe-45b9-83b1-8f495bfc65c0" containerName="revoke-keys"
Oct 30 01:03:28.816422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.816086    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d47ab9-6401-40f8-9533-756429e691f9" containerName="istio-proxy"
Oct 30 01:03:28.816422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.816110    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f565644-a0ed-430d-bc62-e3bac1584495" containerName="payload-processing"
Oct 30 01:03:28.816422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.816121    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae55dd8-21c4-402f-80b9-0d9d91ff9cd6" containerName="istio-proxy"
Oct 30 01:03:28.816422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.816131    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="027a45ec-0dc6-4960-ae70-f66a54134342" containerName="payload-processing"
Oct 30 01:03:28.816422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.816145    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50167c9-f5fe-45b9-83b1-8f495bfc65c0" containerName="revoke-keys"
Oct 30 01:03:28.816422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.816159    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fc1b38-32cc-4853-824e-f3938c66e86e" containerName="payload-pre-processing"
Oct 30 01:03:28.821007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.820985    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.837934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.837872    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"]
Oct 30 01:03:28.843268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.843237    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:28.843432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.843286    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:28.845857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.845834    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:28.849574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.849545    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:03:28.859561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.859533    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/79086be2-22c0-4ae7-ae65-53e8eae7ded0-kube-api-access-rfxhq\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.859741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.859713    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.859816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.859794    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.961570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.961536    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/79086be2-22c0-4ae7-ae65-53e8eae7ded0-kube-api-access-rfxhq\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.961771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.961709    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.961842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.961785    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.962545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.962525    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.964186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.964157    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:28.979984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:28.979958    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/79086be2-22c0-4ae7-ae65-53e8eae7ded0-kube-api-access-rfxhq\") pod \"maas-api-revoke-keys-e2e-migrate-837d7038-jqszk\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:29.137578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:29.137498    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:29.480710 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:03:29.479923    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79086be2_22c0_4ae7_ae65_53e8eae7ded0.slice/crio-89649d0490d0785822320554b0a481747c1cc21db9152b1e4fc62a7fd9ab821b WatchSource:0}: Error finding container 89649d0490d0785822320554b0a481747c1cc21db9152b1e4fc62a7fd9ab821b: Status 404 returned error can't find the container with id 89649d0490d0785822320554b0a481747c1cc21db9152b1e4fc62a7fd9ab821b
Oct 30 01:03:29.480935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:29.480853    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"]
Oct 30 01:03:29.707183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:29.707152    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk" event={"ID":"79086be2-22c0-4ae7-ae65-53e8eae7ded0","Type":"ContainerStarted","Data":"89649d0490d0785822320554b0a481747c1cc21db9152b1e4fc62a7fd9ab821b"}
Oct 30 01:03:30.718417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:30.718382    5104 generic.go:356] "Generic (PLEG): container finished" podID="79086be2-22c0-4ae7-ae65-53e8eae7ded0" containerID="73285bc54a5a17c3c5f30afa8483452e1346c0635e434fb40fa3f49022313a9c" exitCode=0
Oct 30 01:03:30.718847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:30.718509    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk" event={"ID":"79086be2-22c0-4ae7-ae65-53e8eae7ded0","Type":"ContainerDied","Data":"73285bc54a5a17c3c5f30afa8483452e1346c0635e434fb40fa3f49022313a9c"}
Oct 30 01:03:31.027248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:31.027170    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"]
Oct 30 01:03:31.032336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:31.032311    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efbbaded-nrs4s"]
Oct 30 01:03:31.914011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:31.913975    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:32.019015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.018917    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-openshift-ca\") pod \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") "
Oct 30 01:03:32.019015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.018990    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79086be2-22c0-4ae7-ae65-53e8eae7ded0-kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/79086be2-22c0-4ae7-ae65-53e8eae7ded0-kube-api-access-rfxhq\") pod \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") "
Oct 30 01:03:32.019265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.019111    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-xks-ca\") pod \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\" (UID: \"79086be2-22c0-4ae7-ae65-53e8eae7ded0\") "
Oct 30 01:03:32.019407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.019375    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-openshift-ca" pod "79086be2-22c0-4ae7-ae65-53e8eae7ded0" (UID: "79086be2-22c0-4ae7-ae65-53e8eae7ded0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:32.019777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.019749    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:32.021570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.021543    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-xks-ca" pod "79086be2-22c0-4ae7-ae65-53e8eae7ded0" (UID: "79086be2-22c0-4ae7-ae65-53e8eae7ded0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:32.022173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.022148    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79086be2-22c0-4ae7-ae65-53e8eae7ded0-kube-api-access-rfxhq" pod "79086be2-22c0-4ae7-ae65-53e8eae7ded0" (UID: "79086be2-22c0-4ae7-ae65-53e8eae7ded0"). InnerVolumeSpecName "kube-api-access-rfxhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:32.121600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.121559    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/79086be2-22c0-4ae7-ae65-53e8eae7ded0-kube-api-access-rfxhq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:32.121600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.121592    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79086be2-22c0-4ae7-ae65-53e8eae7ded0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:32.581034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.580997    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1482d1-eb55-4458-89ca-b766d00d313a" path="/var/lib/kubelet/pods/ef1482d1-eb55-4458-89ca-b766d00d313a/volumes"
Oct 30 01:03:32.800630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.797780    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"
Oct 30 01:03:32.801397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.801246    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk" event={"ID":"79086be2-22c0-4ae7-ae65-53e8eae7ded0","Type":"ContainerDied","Data":"89649d0490d0785822320554b0a481747c1cc21db9152b1e4fc62a7fd9ab821b"}
Oct 30 01:03:32.803108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:32.803081    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89649d0490d0785822320554b0a481747c1cc21db9152b1e4fc62a7fd9ab821b"
Oct 30 01:03:35.040587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.040552    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"]
Oct 30 01:03:35.041444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.041421    5104 cpu_manager.go:414] "containerMap: removing container" podUID="79086be2-22c0-4ae7-ae65-53e8eae7ded0" containerName="revoke-keys"
Oct 30 01:03:35.041615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.041448    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79086be2-22c0-4ae7-ae65-53e8eae7ded0" containerName="revoke-keys"
Oct 30 01:03:35.041741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.041725    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="79086be2-22c0-4ae7-ae65-53e8eae7ded0" containerName="revoke-keys"
Oct 30 01:03:35.047266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.047243    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.068088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.067889    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:35.068088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.067909    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:35.068942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.068579    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:35.068942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.068649    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:03:35.069697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.069095    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"]
Oct 30 01:03:35.191435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.191383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hnm\" (UniqueName: \"kubernetes.io/projected/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-kube-api-access-86hnm\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.191646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.191555    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.191764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.191741    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.293577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.293477    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hnm\" (UniqueName: \"kubernetes.io/projected/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-kube-api-access-86hnm\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.293743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.293587    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.293743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.293683    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.294471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.294424    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.296147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.296119    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.311383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.311351    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hnm\" (UniqueName: \"kubernetes.io/projected/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-kube-api-access-86hnm\") pod \"maas-api-revoke-sub-ada9711d-cfhkx\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.364437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.364396    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:35.678309 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:03:35.678274    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e6acec_5c73_4257_a7bb_90ebf9a2dd52.slice/crio-ec299f0c02eeb67a7980443424b856a14152cd491bde5213fa22aa1f1f9c305c WatchSource:0}: Error finding container ec299f0c02eeb67a7980443424b856a14152cd491bde5213fa22aa1f1f9c305c: Status 404 returned error can't find the container with id ec299f0c02eeb67a7980443424b856a14152cd491bde5213fa22aa1f1f9c305c
Oct 30 01:03:35.678759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.678736    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"]
Oct 30 01:03:35.834777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:35.834742    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx" event={"ID":"61e6acec-5c73-4257-a7bb-90ebf9a2dd52","Type":"ContainerStarted","Data":"ec299f0c02eeb67a7980443424b856a14152cd491bde5213fa22aa1f1f9c305c"}
Oct 30 01:03:36.847967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:36.847930    5104 generic.go:356] "Generic (PLEG): container finished" podID="61e6acec-5c73-4257-a7bb-90ebf9a2dd52" containerID="653ece8b3b7e713cbfb2393574fb77065407ad122c661f198fa5df82760ce0ff" exitCode=0
Oct 30 01:03:36.848400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:36.847975    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx" event={"ID":"61e6acec-5c73-4257-a7bb-90ebf9a2dd52","Type":"ContainerDied","Data":"653ece8b3b7e713cbfb2393574fb77065407ad122c661f198fa5df82760ce0ff"}
Oct 30 01:03:38.192130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.192103    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:38.365086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.365045    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-kube-api-access-86hnm\" (UniqueName: \"kubernetes.io/projected/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-kube-api-access-86hnm\") pod \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") "
Oct 30 01:03:38.365286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.365150    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-openshift-ca\") pod \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") "
Oct 30 01:03:38.365383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.365366    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-xks-ca\") pod \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\" (UID: \"61e6acec-5c73-4257-a7bb-90ebf9a2dd52\") "
Oct 30 01:03:38.365579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.365545    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-openshift-ca" pod "61e6acec-5c73-4257-a7bb-90ebf9a2dd52" (UID: "61e6acec-5c73-4257-a7bb-90ebf9a2dd52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:38.366215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.366185    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:38.367551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.367520    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-kube-api-access-86hnm" pod "61e6acec-5c73-4257-a7bb-90ebf9a2dd52" (UID: "61e6acec-5c73-4257-a7bb-90ebf9a2dd52"). InnerVolumeSpecName "kube-api-access-86hnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:38.367741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.367717    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-xks-ca" pod "61e6acec-5c73-4257-a7bb-90ebf9a2dd52" (UID: "61e6acec-5c73-4257-a7bb-90ebf9a2dd52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:38.467941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.467910    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:38.467941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.467936    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hnm\" (UniqueName: \"kubernetes.io/projected/61e6acec-5c73-4257-a7bb-90ebf9a2dd52-kube-api-access-86hnm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:38.868676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.868584    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"
Oct 30 01:03:38.868676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.868584    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx" event={"ID":"61e6acec-5c73-4257-a7bb-90ebf9a2dd52","Type":"ContainerDied","Data":"ec299f0c02eeb67a7980443424b856a14152cd491bde5213fa22aa1f1f9c305c"}
Oct 30 01:03:38.868879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:38.868723    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec299f0c02eeb67a7980443424b856a14152cd491bde5213fa22aa1f1f9c305c"
Oct 30 01:03:39.085365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.085330    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"]
Oct 30 01:03:39.086179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.086159    5104 cpu_manager.go:414] "containerMap: removing container" podUID="61e6acec-5c73-4257-a7bb-90ebf9a2dd52" containerName="revoke-keys"
Oct 30 01:03:39.086179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.086178    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e6acec-5c73-4257-a7bb-90ebf9a2dd52" containerName="revoke-keys"
Oct 30 01:03:39.086355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.086332    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e6acec-5c73-4257-a7bb-90ebf9a2dd52" containerName="revoke-keys"
Oct 30 01:03:39.090331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.090306    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.104746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.104719    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:39.112935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.112909    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:39.112935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.112916    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:39.113139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.113028    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:03:39.113613 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:03:39.113575    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:03:39.115121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.115090    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"]
Oct 30 01:03:39.178912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.178877    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bpr\" (UniqueName: \"kubernetes.io/projected/772ec5e3-3c34-4742-8384-bcdee2872a20-kube-api-access-97bpr\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.178912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.178918    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.179205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.179170    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.280523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.280483    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bpr\" (UniqueName: \"kubernetes.io/projected/772ec5e3-3c34-4742-8384-bcdee2872a20-kube-api-access-97bpr\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.281025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.280538    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.281025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.280764    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.281378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.281353    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.284314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.284284    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.299568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.299531    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bpr\" (UniqueName: \"kubernetes.io/projected/772ec5e3-3c34-4742-8384-bcdee2872a20-kube-api-access-97bpr\") pod \"maas-api-revoke-sub-bbbce7c5-gxk7s\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.411552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:39.410382    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:39.987499 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:03:39.986648    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772ec5e3_3c34_4742_8384_bcdee2872a20.slice/crio-ca1d8715a4a24903ad4cbbcdfef7431beec2f603e431647bf3afeb0a66987858 WatchSource:0}: Error finding container ca1d8715a4a24903ad4cbbcdfef7431beec2f603e431647bf3afeb0a66987858: Status 404 returned error can't find the container with id ca1d8715a4a24903ad4cbbcdfef7431beec2f603e431647bf3afeb0a66987858
Oct 30 01:03:40.051955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:40.030563    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"]
Oct 30 01:03:40.893616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:40.893568    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s" event={"ID":"772ec5e3-3c34-4742-8384-bcdee2872a20","Type":"ContainerStarted","Data":"ee739a4cdf6031bb005574cc760d32066df5d9622b262eb15e4f49255b3c54df"}
Oct 30 01:03:40.894092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:40.893631    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s" event={"ID":"772ec5e3-3c34-4742-8384-bcdee2872a20","Type":"ContainerStarted","Data":"ca1d8715a4a24903ad4cbbcdfef7431beec2f603e431647bf3afeb0a66987858"}
Oct 30 01:03:40.930733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:40.930677    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s" podStartSLOduration=1.930663001 podStartE2EDuration="1.930663001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:40.927951359 +0000 UTC m=+1262.475641854" watchObservedRunningTime="2026-10-30 01:03:40.930663001 +0000 UTC m=+1262.478353524"
Oct 30 01:03:41.030656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:41.030622    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"]
Oct 30 01:03:41.036022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:41.035987    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a9164c-t9vbh"]
Oct 30 01:03:41.908080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:41.908043    5104 generic.go:356] "Generic (PLEG): container finished" podID="772ec5e3-3c34-4742-8384-bcdee2872a20" containerID="ee739a4cdf6031bb005574cc760d32066df5d9622b262eb15e4f49255b3c54df" exitCode=0
Oct 30 01:03:41.908537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:41.908085    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s" event={"ID":"772ec5e3-3c34-4742-8384-bcdee2872a20","Type":"ContainerDied","Data":"ee739a4cdf6031bb005574cc760d32066df5d9622b262eb15e4f49255b3c54df"}
Oct 30 01:03:42.580986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:42.580934    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3fc3af6-5316-49a3-9165-dd46f86ac75a" path="/var/lib/kubelet/pods/f3fc3af6-5316-49a3-9165-dd46f86ac75a/volumes"
Oct 30 01:03:43.086907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.086881    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:43.126335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.126296    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-openshift-ca\") pod \"772ec5e3-3c34-4742-8384-bcdee2872a20\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") "
Oct 30 01:03:43.126576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.126520    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-xks-ca\") pod \"772ec5e3-3c34-4742-8384-bcdee2872a20\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") "
Oct 30 01:03:43.126659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.126574    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772ec5e3-3c34-4742-8384-bcdee2872a20-kube-api-access-97bpr\" (UniqueName: \"kubernetes.io/projected/772ec5e3-3c34-4742-8384-bcdee2872a20-kube-api-access-97bpr\") pod \"772ec5e3-3c34-4742-8384-bcdee2872a20\" (UID: \"772ec5e3-3c34-4742-8384-bcdee2872a20\") "
Oct 30 01:03:43.126793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.126760    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-openshift-ca" pod "772ec5e3-3c34-4742-8384-bcdee2872a20" (UID: "772ec5e3-3c34-4742-8384-bcdee2872a20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:43.127303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.127277    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:43.129916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.129889    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-xks-ca" pod "772ec5e3-3c34-4742-8384-bcdee2872a20" (UID: "772ec5e3-3c34-4742-8384-bcdee2872a20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:43.130011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.129944    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772ec5e3-3c34-4742-8384-bcdee2872a20-kube-api-access-97bpr" pod "772ec5e3-3c34-4742-8384-bcdee2872a20" (UID: "772ec5e3-3c34-4742-8384-bcdee2872a20"). InnerVolumeSpecName "kube-api-access-97bpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:43.228412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.228374    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772ec5e3-3c34-4742-8384-bcdee2872a20-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:43.228412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.228407    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97bpr\" (UniqueName: \"kubernetes.io/projected/772ec5e3-3c34-4742-8384-bcdee2872a20-kube-api-access-97bpr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:43.933398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.933364    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"
Oct 30 01:03:43.933759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.933363    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s" event={"ID":"772ec5e3-3c34-4742-8384-bcdee2872a20","Type":"ContainerDied","Data":"ca1d8715a4a24903ad4cbbcdfef7431beec2f603e431647bf3afeb0a66987858"}
Oct 30 01:03:43.933759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:43.933517    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1d8715a4a24903ad4cbbcdfef7431beec2f603e431647bf3afeb0a66987858"
Oct 30 01:03:45.367539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.367483    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"]
Oct 30 01:03:45.368512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.368487    5104 cpu_manager.go:414] "containerMap: removing container" podUID="772ec5e3-3c34-4742-8384-bcdee2872a20" containerName="revoke-keys"
Oct 30 01:03:45.368512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.368512    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772ec5e3-3c34-4742-8384-bcdee2872a20" containerName="revoke-keys"
Oct 30 01:03:45.368744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.368726    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="772ec5e3-3c34-4742-8384-bcdee2872a20" containerName="revoke-keys"
Oct 30 01:03:45.373005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.372982    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.391630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.391599    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"]
Oct 30 01:03:45.394090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.394060    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:45.394244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.394102    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:03:45.394244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.394151    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:45.394374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.394296    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:45.471030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.470996    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxbwz\" (UniqueName: \"kubernetes.io/projected/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-kube-api-access-gxbwz\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.471241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.471209    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.471379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.471362    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.572839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.572802    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxbwz\" (UniqueName: \"kubernetes.io/projected/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-kube-api-access-gxbwz\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.573052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.572957    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.573360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.573330    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.573792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.573769    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.575685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.575659    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.591493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.591468    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxbwz\" (UniqueName: \"kubernetes.io/projected/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-kube-api-access-gxbwz\") pod \"maas-api-revoke-sub-5709a163-f72v4\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.693617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.693587    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:45.996780 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:03:45.996747    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7456860a_d5c7_4b4d_ae07_f8ff42129f7b.slice/crio-8813c4775a7fcd6b138ba422408888c1f12068425972c4102aa9f79cfd014b72 WatchSource:0}: Error finding container 8813c4775a7fcd6b138ba422408888c1f12068425972c4102aa9f79cfd014b72: Status 404 returned error can't find the container with id 8813c4775a7fcd6b138ba422408888c1f12068425972c4102aa9f79cfd014b72
Oct 30 01:03:45.996978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:45.996959    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"]
Oct 30 01:03:46.964200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:46.964168    5104 generic.go:356] "Generic (PLEG): container finished" podID="7456860a-d5c7-4b4d-ae07-f8ff42129f7b" containerID="fd042e985e18021d2cd7f14037efa41c4d04ed4c031023dd7fefeb84ff3691c3" exitCode=0
Oct 30 01:03:46.964663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:46.964251    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4" event={"ID":"7456860a-d5c7-4b4d-ae07-f8ff42129f7b","Type":"ContainerDied","Data":"fd042e985e18021d2cd7f14037efa41c4d04ed4c031023dd7fefeb84ff3691c3"}
Oct 30 01:03:46.964663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:46.964303    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4" event={"ID":"7456860a-d5c7-4b4d-ae07-f8ff42129f7b","Type":"ContainerStarted","Data":"8813c4775a7fcd6b138ba422408888c1f12068425972c4102aa9f79cfd014b72"}
Oct 30 01:03:48.175083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.175057    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:48.230758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.230725    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-kube-api-access-gxbwz\" (UniqueName: \"kubernetes.io/projected/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-kube-api-access-gxbwz\") pod \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") "
Oct 30 01:03:48.230921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.230839    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-openshift-ca\") pod \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") "
Oct 30 01:03:48.231052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.230926    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-xks-ca\") pod \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\" (UID: \"7456860a-d5c7-4b4d-ae07-f8ff42129f7b\") "
Oct 30 01:03:48.231286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.231248    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-openshift-ca" pod "7456860a-d5c7-4b4d-ae07-f8ff42129f7b" (UID: "7456860a-d5c7-4b4d-ae07-f8ff42129f7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:48.231662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.231640    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:48.233620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.233593    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-kube-api-access-gxbwz" pod "7456860a-d5c7-4b4d-ae07-f8ff42129f7b" (UID: "7456860a-d5c7-4b4d-ae07-f8ff42129f7b"). InnerVolumeSpecName "kube-api-access-gxbwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:48.233719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.233636    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-xks-ca" pod "7456860a-d5c7-4b4d-ae07-f8ff42129f7b" (UID: "7456860a-d5c7-4b4d-ae07-f8ff42129f7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:48.333450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.333419    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:48.333450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.333446    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxbwz\" (UniqueName: \"kubernetes.io/projected/7456860a-d5c7-4b4d-ae07-f8ff42129f7b-kube-api-access-gxbwz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:48.998809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.998775    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"
Oct 30 01:03:48.998809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.998799    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4" event={"ID":"7456860a-d5c7-4b4d-ae07-f8ff42129f7b","Type":"ContainerDied","Data":"8813c4775a7fcd6b138ba422408888c1f12068425972c4102aa9f79cfd014b72"}
Oct 30 01:03:48.999082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:48.998870    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8813c4775a7fcd6b138ba422408888c1f12068425972c4102aa9f79cfd014b72"
Oct 30 01:03:50.520416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.520380    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"]
Oct 30 01:03:50.521517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.521495    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7456860a-d5c7-4b4d-ae07-f8ff42129f7b" containerName="revoke-keys"
Oct 30 01:03:50.521638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.521518    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7456860a-d5c7-4b4d-ae07-f8ff42129f7b" containerName="revoke-keys"
Oct 30 01:03:50.521773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.521760    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7456860a-d5c7-4b4d-ae07-f8ff42129f7b" containerName="revoke-keys"
Oct 30 01:03:50.526667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.526639    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.548066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.547990    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-18a189-openshift-default-dockercfg-rcm79\""
Oct 30 01:03:50.565568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.565539    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"]
Oct 30 01:03:50.683174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683137    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.683362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683192    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93bf74-fd15-47c7-8472-401b203e726e-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.683362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683222    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5wk\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-kube-api-access-kx5wk\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.683491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683363    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.683556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683533    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.683630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683605    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.683870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683846    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93bf74-fd15-47c7-8472-401b203e726e-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.683953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.683935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.684074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.684060    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.787622    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93bf74-fd15-47c7-8472-401b203e726e-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.787709    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.787858    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.788025    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.788065    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93bf74-fd15-47c7-8472-401b203e726e-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.788096    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5wk\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-kube-api-access-kx5wk\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.788166    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.788230    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.788943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.788284    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.790943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.790920    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93bf74-fd15-47c7-8472-401b203e726e-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.792019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.791980    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.808509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.794741    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.808509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.798615    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.809600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.809000    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.827266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.826700    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.827266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.827203    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.828200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.828148    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5wk\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-kube-api-access-kx5wk\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:50.868482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:50.859594    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93bf74-fd15-47c7-8472-401b203e726e-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:51.146328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:51.146245    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:51.511338 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:03:51.511305    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa93bf74_fd15_47c7_8472_401b203e726e.slice/crio-91b2954f06a9960144cdfdd64967b6aed8ae2ca072c48b5a52ac968bf3b2b64a WatchSource:0}: Error finding container 91b2954f06a9960144cdfdd64967b6aed8ae2ca072c48b5a52ac968bf3b2b64a: Status 404 returned error can't find the container with id 91b2954f06a9960144cdfdd64967b6aed8ae2ca072c48b5a52ac968bf3b2b64a
Oct 30 01:03:51.512985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:51.512959    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"]
Oct 30 01:03:51.514831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:51.514790    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:03:51.514913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:51.514878    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:03:51.514966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:51.514920    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:03:52.118916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:52.118874    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" event={"ID":"fa93bf74-fd15-47c7-8472-401b203e726e","Type":"ContainerStarted","Data":"9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371"}
Oct 30 01:03:52.119422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:52.118943    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" event={"ID":"fa93bf74-fd15-47c7-8472-401b203e726e","Type":"ContainerStarted","Data":"91b2954f06a9960144cdfdd64967b6aed8ae2ca072c48b5a52ac968bf3b2b64a"}
Oct 30 01:03:52.120852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:52.120829    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"]
Oct 30 01:03:52.121012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:52.120993    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy" containerID="cri-o://4af27eadd7e3c4ce1f9ec3a8ff54a30ca9ab22add19f56cf200279cd64dae889" gracePeriod=30
Oct 30 01:03:52.147487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:52.147406    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:52.154960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:52.154880    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:52.159061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:52.159004    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podStartSLOduration=2.158985318 podStartE2EDuration="2.158985318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:52.156153007 +0000 UTC m=+1273.703843503" watchObservedRunningTime="2026-10-30 01:03:52.158985318 +0000 UTC m=+1273.706675815"
Oct 30 01:03:53.161698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:53.161659    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:53.163497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:53.163428    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:03:53.509083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:53.509037    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"]
Oct 30 01:03:53.517561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:53.517521    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-837d7038-jqszk"]
Oct 30 01:03:54.580185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:54.580149    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79086be2-22c0-4ae7-ae65-53e8eae7ded0" path="/var/lib/kubelet/pods/79086be2-22c0-4ae7-ae65-53e8eae7ded0/volumes"
Oct 30 01:03:55.177896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.177861    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"]
Oct 30 01:03:55.185827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.185796    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.199975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.199950    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-data\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.200293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.200265    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b90433-33b5-49ee-9cd5-2faf0655badd-istiod-ca-cert\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.201250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.201230    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-certs\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.201470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.201439    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-token\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.202118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.202099    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-envoy\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.202259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.202240    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldw5q\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-kube-api-access-ldw5q\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.202445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.202423    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-credential-socket\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.202590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.202532    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-podinfo\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.202653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.202594    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-socket\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.209756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.209733    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"]
Oct 30 01:03:55.214402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.212697    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-08065c8e-openshift-default-dockercfg-8wxxn\""
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341003    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-token\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341064    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-envoy\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341096    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldw5q\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-kube-api-access-ldw5q\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341186    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-credential-socket\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341242    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-podinfo\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341307    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-socket\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341399    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-data\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341552    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b90433-33b5-49ee-9cd5-2faf0655badd-istiod-ca-cert\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.341596    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-certs\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.342216    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-credential-socket\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.344408    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b90433-33b5-49ee-9cd5-2faf0655badd-istiod-ca-cert\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.345704    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-data\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.346484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.345907    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-socket\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.347566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.346672    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-certs\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.412482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.411763    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-envoy\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.412482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.412360    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-podinfo\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.479554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.475226    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldw5q\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-kube-api-access-ldw5q\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.492302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.492230    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-token\") pod \"e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") " pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:55.530049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:55.529752    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:56.417409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:56.417363    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"]
Oct 30 01:03:56.531114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:56.529867    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:03:56.531114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:56.529971    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:03:56.531114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:56.530018    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:03:57.484490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.471102    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"]
Oct 30 01:03:57.518600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.516011    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:57.560592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.539420    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt5s9\" (UniqueName: \"kubernetes.io/projected/1bac07a8-d189-4a85-8961-7414d74492ce-kube-api-access-rt5s9\") pod \"payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:57.560592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.539674    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac07a8-d189-4a85-8961-7414d74492ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:57.605523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.590634    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-18a189\""
Oct 30 01:03:57.648480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.641583    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac07a8-d189-4a85-8961-7414d74492ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:57.648480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.641997    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt5s9\" (UniqueName: \"kubernetes.io/projected/1bac07a8-d189-4a85-8961-7414d74492ce-kube-api-access-rt5s9\") pod \"payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:57.648480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.643528    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac07a8-d189-4a85-8961-7414d74492ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:57.727376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.711952    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-18a189-dockercfg-xbmxx\""
Oct 30 01:03:57.794577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.786630    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt5s9\" (UniqueName: \"kubernetes.io/projected/1bac07a8-d189-4a85-8961-7414d74492ce-kube-api-access-rt5s9\") pod \"payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:57.987475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.981665    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" event={"ID":"84b90433-33b5-49ee-9cd5-2faf0655badd","Type":"ContainerStarted","Data":"4c3ee4fad64fa2c9c765525dc9a6cd2b2355a9432c98a0af326ec7d4a9c115e8"}
Oct 30 01:03:57.987475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.981744    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" event={"ID":"84b90433-33b5-49ee-9cd5-2faf0655badd","Type":"ContainerStarted","Data":"8c2e90c012b7b315829540e269e3c81239da9b37019844fe22d2f136270920cf"}
Oct 30 01:03:58.018476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.013518    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:03:58.061727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:57.812566    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:03:58.116629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.114780    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:03:58.194724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.194658    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"]
Oct 30 01:03:58.368996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.366587    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:03:58.396535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.389541    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"]
Oct 30 01:03:58.443519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.438789    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.477736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.477701    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-eacea3-openshift-default-dockercfg-bqqn7\""
Oct 30 01:03:58.569864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.548260    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.569864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.548361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.569864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.548397    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.569864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.554722    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:58.616479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.597986    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.616479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.598222    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.616479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.598274    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.616479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.598380    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6sc\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-kube-api-access-zv6sc\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.616479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.598427    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.616479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.598504    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.616479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.608263    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:03:58.642023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.641909    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podStartSLOduration=3.641885996 podStartE2EDuration="3.641885996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:58.096273512 +0000 UTC m=+1279.643964009" watchObservedRunningTime="2026-10-30 01:03:58.641885996 +0000 UTC m=+1280.189576494"
Oct 30 01:03:58.708482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.683410    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"]
Oct 30 01:03:58.727758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.727728    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:58.728384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.700324    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.728744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.728718    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.728905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.728884    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.729198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.729097    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.729445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.729422    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.729626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.729602    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.729833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.729806    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6sc\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-kube-api-access-zv6sc\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.729972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.729951    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.730130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.730109    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.731182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.731141    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.731970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.731935    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.736978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.736346    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.737091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.736984    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.740211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.737591    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.740211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.738528    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.740211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.739210    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.876228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.876188    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6sc\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-kube-api-access-zv6sc\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:58.921530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:58.916158    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:59.232327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.232244    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:03:59.616491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.613773    5104 generic.go:356] "Generic (PLEG): container finished" podID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerID="4af27eadd7e3c4ce1f9ec3a8ff54a30ca9ab22add19f56cf200279cd64dae889" exitCode=0
Oct 30 01:03:59.625842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.625798    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" event={"ID":"0c7221b2-0f22-4fe0-8db3-87fba47674af","Type":"ContainerDied","Data":"4af27eadd7e3c4ce1f9ec3a8ff54a30ca9ab22add19f56cf200279cd64dae889"}
Oct 30 01:03:59.638116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.637966    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 01:03:59.640087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.640059    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:59.677543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.677500    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"]
Oct 30 01:03:59.681885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.681745    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:03:59.812985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.812950    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-certs\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813035    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-podinfo\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813103    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7221b2-0f22-4fe0-8db3-87fba47674af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7221b2-0f22-4fe0-8db3-87fba47674af-istiod-ca-cert\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813201    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-socket\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813262    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-credential-socket\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813316    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-data\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813377    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-envoy\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813413    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-kube-api-access-b5bc5\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-kube-api-access-b5bc5\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.813512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.813444    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-token\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-token\") pod \"0c7221b2-0f22-4fe0-8db3-87fba47674af\" (UID: \"0c7221b2-0f22-4fe0-8db3-87fba47674af\") "
Oct 30 01:03:59.814936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.814759    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-socket" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:59.815307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.815143    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7221b2-0f22-4fe0-8db3-87fba47674af-istiod-ca-cert" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:59.815551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.815396    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-certs" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:59.815933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.815688    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-data" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:59.815933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.815742    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-credential-socket" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:59.823520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.823401    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-kube-api-access-b5bc5" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "kube-api-access-b5bc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:59.824131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.824089    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-envoy" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:59.827961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.827936    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-token" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:59.829746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.829724    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-podinfo" pod "0c7221b2-0f22-4fe0-8db3-87fba47674af" (UID: "0c7221b2-0f22-4fe0-8db3-87fba47674af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:59.915953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.915911    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.915953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.915952    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.916229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.915967    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.916229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.915980    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5bc5\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-kube-api-access-b5bc5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.916229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.915994    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.916229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.916007    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.916229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.916019    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7221b2-0f22-4fe0-8db3-87fba47674af-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.916229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.916031    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7221b2-0f22-4fe0-8db3-87fba47674af-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:03:59.916229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:03:59.916044    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7221b2-0f22-4fe0-8db3-87fba47674af-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:04:00.006641 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:00.006604    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc081e5d8_e6c1_4739_8a2f_a363e06b8e12.slice/crio-90e9e2c464cbd53e4bf377c56af1a17f7c4e11290f4294a2ca3d8d3412fe6f23 WatchSource:0}: Error finding container 90e9e2c464cbd53e4bf377c56af1a17f7c4e11290f4294a2ca3d8d3412fe6f23: Status 404 returned error can't find the container with id 90e9e2c464cbd53e4bf377c56af1a17f7c4e11290f4294a2ca3d8d3412fe6f23
Oct 30 01:04:00.009043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.009012    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"]
Oct 30 01:04:00.009505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.009472    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:00.009562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.009542    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:00.009598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.009573    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:00.625141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.625112    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"
Oct 30 01:04:00.625594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.625143    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l" event={"ID":"0c7221b2-0f22-4fe0-8db3-87fba47674af","Type":"ContainerDied","Data":"7db1c3cd45e01f49f66539f46d9e690211a8d0948b07bbf1333510897ff26cbd"}
Oct 30 01:04:00.625594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.625206    5104 scope.go:122] "RemoveContainer" containerID="4af27eadd7e3c4ce1f9ec3a8ff54a30ca9ab22add19f56cf200279cd64dae889"
Oct 30 01:04:00.627801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.627777    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" event={"ID":"c081e5d8-e6c1-4739-8a2f-a363e06b8e12","Type":"ContainerStarted","Data":"818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0"}
Oct 30 01:04:00.627918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.627829    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" event={"ID":"c081e5d8-e6c1-4739-8a2f-a363e06b8e12","Type":"ContainerStarted","Data":"90e9e2c464cbd53e4bf377c56af1a17f7c4e11290f4294a2ca3d8d3412fe6f23"}
Oct 30 01:04:00.629691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.629654    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" event={"ID":"1bac07a8-d189-4a85-8961-7414d74492ce","Type":"ContainerStarted","Data":"a0b2c60915418659922bd9dbf7d65f112d494ea003b010bd2e0f47c7c5f8224c"}
Oct 30 01:04:00.629789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.629700    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" event={"ID":"1bac07a8-d189-4a85-8961-7414d74492ce","Type":"ContainerStarted","Data":"e2de0801ad5b57e6445ab7530870745ec9b7d42779e86b0c3ccd486701f1d930"}
Oct 30 01:04:00.629908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.629888    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:04:00.692502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.692451    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"]
Oct 30 01:04:00.698292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.698264    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-w5z5l"]
Oct 30 01:04:00.734451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.734393    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" podStartSLOduration=3.7343800700000003 podStartE2EDuration="3.73438007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:00.730114082 +0000 UTC m=+1282.277804578" watchObservedRunningTime="2026-10-30 01:04:00.73438007 +0000 UTC m=+1282.282070566"
Oct 30 01:04:00.778159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:00.778104    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podStartSLOduration=3.77809042 podStartE2EDuration="3.77809042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:00.773512362 +0000 UTC m=+1282.321202857" watchObservedRunningTime="2026-10-30 01:04:00.77809042 +0000 UTC m=+1282.325780917"
Oct 30 01:04:01.234535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:01.233080    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:04:01.239999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:01.239957    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:04:01.643614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:01.643509    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:04:01.644369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:01.644347    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:04:02.583067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.583032    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" path="/var/lib/kubelet/pods/0c7221b2-0f22-4fe0-8db3-87fba47674af/volumes"
Oct 30 01:04:02.599684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.599641    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"]
Oct 30 01:04:02.600813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.600789    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy"
Oct 30 01:04:02.600813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.600812    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy"
Oct 30 01:04:02.601074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.601060    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7221b2-0f22-4fe0-8db3-87fba47674af" containerName="istio-proxy"
Oct 30 01:04:02.611808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.608772    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:02.632676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.632651    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-08065c8e-dockercfg-np5fh\""
Oct 30 01:04:02.632845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.632823    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-08065c8e\""
Oct 30 01:04:02.637842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.637810    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"]
Oct 30 01:04:02.700646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.700613    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"]
Oct 30 01:04:02.706054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.706025    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:02.727108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.727069    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"]
Oct 30 01:04:02.854536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.795442    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89g6\" (UniqueName: \"kubernetes.io/projected/73189491-ceaf-469c-951c-f22773bd9c23-kube-api-access-m89g6\") pod \"payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") " pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:02.884482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.855133    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73189491-ceaf-469c-951c-f22773bd9c23-plugins-config-volume\") pod \"payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") " pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:02.888480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.884902    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtmq\" (UniqueName: \"kubernetes.io/projected/2189dd1e-6efe-408d-88ca-ae10099a0097-kube-api-access-4jtmq\") pod \"payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:02.913696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:02.913662    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2189dd1e-6efe-408d-88ca-ae10099a0097-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:03.117594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.117417    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89g6\" (UniqueName: \"kubernetes.io/projected/73189491-ceaf-469c-951c-f22773bd9c23-kube-api-access-m89g6\") pod \"payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") " pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:03.118027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.117991    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73189491-ceaf-469c-951c-f22773bd9c23-plugins-config-volume\") pod \"payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") " pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:03.118175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.118108    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtmq\" (UniqueName: \"kubernetes.io/projected/2189dd1e-6efe-408d-88ca-ae10099a0097-kube-api-access-4jtmq\") pod \"payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:03.118175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.118172    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2189dd1e-6efe-408d-88ca-ae10099a0097-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:03.121158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.119305    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2189dd1e-6efe-408d-88ca-ae10099a0097-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:03.122338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.122303    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73189491-ceaf-469c-951c-f22773bd9c23-plugins-config-volume\") pod \"payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") " pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:03.140391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.140361    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtmq\" (UniqueName: \"kubernetes.io/projected/2189dd1e-6efe-408d-88ca-ae10099a0097-kube-api-access-4jtmq\") pod \"payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:03.140391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.140387    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89g6\" (UniqueName: \"kubernetes.io/projected/73189491-ceaf-469c-951c-f22773bd9c23-kube-api-access-m89g6\") pod \"payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") " pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:03.243637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.243607    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:03.339025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.330776    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:03.623929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.623846    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"]
Oct 30 01:04:03.633079 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:03.633031    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2189dd1e_6efe_408d_88ca_ae10099a0097.slice/crio-df17c05dd783c2ad21b5a9311fabec658b421fc0d4b4c9df10cc1460dd67d0d0 WatchSource:0}: Error finding container df17c05dd783c2ad21b5a9311fabec658b421fc0d4b4c9df10cc1460dd67d0d0: Status 404 returned error can't find the container with id df17c05dd783c2ad21b5a9311fabec658b421fc0d4b4c9df10cc1460dd67d0d0
Oct 30 01:04:03.676519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.676491    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds" event={"ID":"2189dd1e-6efe-408d-88ca-ae10099a0097","Type":"ContainerStarted","Data":"df17c05dd783c2ad21b5a9311fabec658b421fc0d4b4c9df10cc1460dd67d0d0"}
Oct 30 01:04:03.774172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:03.774147    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"]
Oct 30 01:04:04.602072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.602040    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"]
Oct 30 01:04:04.608410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.608368    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:04.625424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.625392    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"]
Oct 30 01:04:04.628935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.628902    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fb3370cb-dockercfg-thq5n\""
Oct 30 01:04:04.645244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.645174    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb3370cb\""
Oct 30 01:04:04.666818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.659275    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ld9\" (UniqueName: \"kubernetes.io/projected/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-kube-api-access-k8ld9\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:04.667378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.667344    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:04.805144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.788484    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:04.980494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.977414    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:04.980494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:04.977866    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ld9\" (UniqueName: \"kubernetes.io/projected/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-kube-api-access-k8ld9\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:05.051685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.051640    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds" event={"ID":"2189dd1e-6efe-408d-88ca-ae10099a0097","Type":"ContainerStarted","Data":"7fd5b4357d851893f3162c6619551bc35ba358f1971a134f85bf4b3655ff1d39"}
Oct 30 01:04:05.052140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.052116    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:05.060210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.060176    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ld9\" (UniqueName: \"kubernetes.io/projected/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-kube-api-access-k8ld9\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:05.061083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.061040    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks" event={"ID":"73189491-ceaf-469c-951c-f22773bd9c23","Type":"ContainerStarted","Data":"ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6"}
Oct 30 01:04:05.061222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.061107    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks" event={"ID":"73189491-ceaf-469c-951c-f22773bd9c23","Type":"ContainerStarted","Data":"a9298db5c2cdab96a852d0723bc8f41752e796ebf654bc5212cabc420d748eeb"}
Oct 30 01:04:05.061222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.061146    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:05.089917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.089857    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds" podStartSLOduration=3.089842175 podStartE2EDuration="3.089842175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:05.087769221 +0000 UTC m=+1286.635459716" watchObservedRunningTime="2026-10-30 01:04:05.089842175 +0000 UTC m=+1286.637532671"
Oct 30 01:04:05.131003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.130943    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks" podStartSLOduration=3.130927721 podStartE2EDuration="3.130927721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:05.126222253 +0000 UTC m=+1286.673912750" watchObservedRunningTime="2026-10-30 01:04:05.130927721 +0000 UTC m=+1286.678618216"
Oct 30 01:04:05.229642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:05.229605    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:06.208416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:06.204509    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"]
Oct 30 01:04:07.292480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:07.281564    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" event={"ID":"d708b71f-6a70-4a1d-a8e6-56cc9f56596d","Type":"ContainerStarted","Data":"1faa4c2ff1e6f40e166761025c923be7b751062118b9cedb6afa2d9b5376ccca"}
Oct 30 01:04:07.292480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:07.281634    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" event={"ID":"d708b71f-6a70-4a1d-a8e6-56cc9f56596d","Type":"ContainerStarted","Data":"c6f3624c6a56f7b4d8d7907eaf73d493cef5fd9a680ffbbfc4dca371edfe7005"}
Oct 30 01:04:07.292480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:07.282886    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:07.348608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:07.348559    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" podStartSLOduration=3.348540455 podStartE2EDuration="3.348540455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:07.323853129 +0000 UTC m=+1288.871543625" watchObservedRunningTime="2026-10-30 01:04:07.348540455 +0000 UTC m=+1288.896230950"
Oct 30 01:04:09.120821 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:04:09.120780    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:04:11.646825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:11.646796    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:04:12.041481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:12.041420    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:04:16.220156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.220125    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:04:16.220701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.220201    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:04:16.831246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.831206    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"]
Oct 30 01:04:16.837482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.837441    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:16.853214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.853182    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"]
Oct 30 01:04:16.856923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.856894    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:04:16.857075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.856956    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:04:16.857075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.856963    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:04:16.862998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.862976    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:04:16.954323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.954286    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflpc\" (UniqueName: \"kubernetes.io/projected/b0cb0863-578b-4ac3-b13e-33cb64aca759-kube-api-access-kflpc\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:16.954512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.954412    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:16.954512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:16.954506    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.056327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.056280    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflpc\" (UniqueName: \"kubernetes.io/projected/b0cb0863-578b-4ac3-b13e-33cb64aca759-kube-api-access-kflpc\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.056566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.056420    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.056566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.056494    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.057241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.057220    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.058988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.058963    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.077729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.077692    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflpc\" (UniqueName: \"kubernetes.io/projected/b0cb0863-578b-4ac3-b13e-33cb64aca759-kube-api-access-kflpc\") pod \"maas-api-revoke-sub-6159cce8-qsqln\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.154805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.154723    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:17.486049 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:17.486018    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cb0863_578b_4ac3_b13e_33cb64aca759.slice/crio-dac5f340fad88ea6448ca3198f0535902d87364fc4cef7f28c3fa182c3f80593 WatchSource:0}: Error finding container dac5f340fad88ea6448ca3198f0535902d87364fc4cef7f28c3fa182c3f80593: Status 404 returned error can't find the container with id dac5f340fad88ea6448ca3198f0535902d87364fc4cef7f28c3fa182c3f80593
Oct 30 01:04:17.487374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.487343    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"]
Oct 30 01:04:17.537509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:17.537474    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln" event={"ID":"b0cb0863-578b-4ac3-b13e-33cb64aca759","Type":"ContainerStarted","Data":"dac5f340fad88ea6448ca3198f0535902d87364fc4cef7f28c3fa182c3f80593"}
Oct 30 01:04:18.549623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:18.549590    5104 generic.go:356] "Generic (PLEG): container finished" podID="b0cb0863-578b-4ac3-b13e-33cb64aca759" containerID="e931f770e5bdab744c6acb1061e55c851a5b8691b5d595782882cb4569e33ee5" exitCode=0
Oct 30 01:04:18.550014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:18.549670    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln" event={"ID":"b0cb0863-578b-4ac3-b13e-33cb64aca759","Type":"ContainerDied","Data":"e931f770e5bdab744c6acb1061e55c851a5b8691b5d595782882cb4569e33ee5"}
Oct 30 01:04:19.311849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.311819    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:04:19.760962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.760934    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:19.838628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.838594    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-xks-ca\") pod \"b0cb0863-578b-4ac3-b13e-33cb64aca759\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") "
Oct 30 01:04:19.838812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.838692    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cb0863-578b-4ac3-b13e-33cb64aca759-kube-api-access-kflpc\" (UniqueName: \"kubernetes.io/projected/b0cb0863-578b-4ac3-b13e-33cb64aca759-kube-api-access-kflpc\") pod \"b0cb0863-578b-4ac3-b13e-33cb64aca759\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") "
Oct 30 01:04:19.838812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.838743    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-openshift-ca\") pod \"b0cb0863-578b-4ac3-b13e-33cb64aca759\" (UID: \"b0cb0863-578b-4ac3-b13e-33cb64aca759\") "
Oct 30 01:04:19.839216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.839182    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-openshift-ca" pod "b0cb0863-578b-4ac3-b13e-33cb64aca759" (UID: "b0cb0863-578b-4ac3-b13e-33cb64aca759"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:19.839542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.839523    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:04:19.840878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.840848    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-xks-ca" pod "b0cb0863-578b-4ac3-b13e-33cb64aca759" (UID: "b0cb0863-578b-4ac3-b13e-33cb64aca759"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:19.841301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.841278    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cb0863-578b-4ac3-b13e-33cb64aca759-kube-api-access-kflpc" pod "b0cb0863-578b-4ac3-b13e-33cb64aca759" (UID: "b0cb0863-578b-4ac3-b13e-33cb64aca759"). InnerVolumeSpecName "kube-api-access-kflpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:19.941032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.940995    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cb0863-578b-4ac3-b13e-33cb64aca759-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:04:19.941032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:19.941028    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflpc\" (UniqueName: \"kubernetes.io/projected/b0cb0863-578b-4ac3-b13e-33cb64aca759-kube-api-access-kflpc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:04:20.571063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:20.571035    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"
Oct 30 01:04:20.582273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:20.582240    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln" event={"ID":"b0cb0863-578b-4ac3-b13e-33cb64aca759","Type":"ContainerDied","Data":"dac5f340fad88ea6448ca3198f0535902d87364fc4cef7f28c3fa182c3f80593"}
Oct 30 01:04:20.582405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:20.582281    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac5f340fad88ea6448ca3198f0535902d87364fc4cef7f28c3fa182c3f80593"
Oct 30 01:04:21.039057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.039019    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"]
Oct 30 01:04:21.052990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.052957    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a288023-2kzl6"]
Oct 30 01:04:21.495602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.495555    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"]
Oct 30 01:04:21.497027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.496994    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b0cb0863-578b-4ac3-b13e-33cb64aca759" containerName="revoke-keys"
Oct 30 01:04:21.497027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.497025    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cb0863-578b-4ac3-b13e-33cb64aca759" containerName="revoke-keys"
Oct 30 01:04:21.497224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.497211    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cb0863-578b-4ac3-b13e-33cb64aca759" containerName="revoke-keys"
Oct 30 01:04:21.501547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.501526    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.516130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.516100    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"]
Oct 30 01:04:21.523047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.523019    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.543539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.543506    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"]
Oct 30 01:04:21.575059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.575026    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"]
Oct 30 01:04:21.675052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675012    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61231598-2fc8-4806-97a9-fc167d024f3c-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.675238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675075    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.675238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675185    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.675238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675235    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.675425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675266    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.675515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675416    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.675588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675526    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.675588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675565    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.675688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675612    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.675688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675671    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61231598-2fc8-4806-97a9-fc167d024f3c-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.675775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675727    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd059c20-b2e8-41f5-81d8-19b831922f9e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.675775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675756    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2hj\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-kube-api-access-rv2hj\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.675887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675864    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.675944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675900    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.675944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.675932    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.676064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.676049    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.676120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.676086    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.676245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.676231    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj75z\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-kube-api-access-dj75z\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.777520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777381    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd059c20-b2e8-41f5-81d8-19b831922f9e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.777520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777448    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2hj\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-kube-api-access-rv2hj\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777534    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777566    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777595    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777686    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777717    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777818    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj75z\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-kube-api-access-dj75z\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777892    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61231598-2fc8-4806-97a9-fc167d024f3c-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777938    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.777978    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778013    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778065    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778149    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778188    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778221    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778260    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.778681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778296    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61231598-2fc8-4806-97a9-fc167d024f3c-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.778681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778322    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.779110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778684    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.779110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.778894    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.779240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.779143    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.780626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.779413    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.780626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.779846    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd059c20-b2e8-41f5-81d8-19b831922f9e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.780626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.780089    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.780626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.780257    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61231598-2fc8-4806-97a9-fc167d024f3c-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.780626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.780331    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.780626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.780537    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.781332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.781307    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.781528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.781504    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61231598-2fc8-4806-97a9-fc167d024f3c-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.781650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.781525    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.781705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.781666    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.797008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.796978    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.797571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.797551    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2hj\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-kube-api-access-rv2hj\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.797891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.797867    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.798084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.798060    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj75z\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-kube-api-access-dj75z\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:21.845483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.845439    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:21.871349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:21.871317    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:22.222210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.222178    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"]
Oct 30 01:04:22.229432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.229403    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.252665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.252640    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"]
Oct 30 01:04:22.271179 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:22.271148    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd059c20_b2e8_41f5_81d8_19b831922f9e.slice/crio-63c0dc60eb03e417bd58d3fa03d8d6d64f712b8198190ee3ff0774c98371d3a1 WatchSource:0}: Error finding container 63c0dc60eb03e417bd58d3fa03d8d6d64f712b8198190ee3ff0774c98371d3a1: Status 404 returned error can't find the container with id 63c0dc60eb03e417bd58d3fa03d8d6d64f712b8198190ee3ff0774c98371d3a1
Oct 30 01:04:22.275027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.274575    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:22.275027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.274663    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:22.275027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.274709    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:22.286136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.286116    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"]
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.397952    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398046    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398206    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9114fe-c972-4605-afa7-3975a55d3712-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398270    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9114fe-c972-4605-afa7-3975a55d3712-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398364    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398511    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398702    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lshq\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-kube-api-access-5lshq\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398765    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.406482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.398818    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.426515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.426429    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:22.426668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.426564    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:22.426668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.426612    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:22.509280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.509211    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9114fe-c972-4605-afa7-3975a55d3712-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.511099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.511067    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.511353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.511335    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.511500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.511484    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lshq\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-kube-api-access-5lshq\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.511665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.511649    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.511823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.511804    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.512094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.512069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.512304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.512283    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.512719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.512697    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9114fe-c972-4605-afa7-3975a55d3712-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.513574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.513536    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.513574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.512183    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.513725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.512436    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.513906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.512724    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.516268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.516240    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9114fe-c972-4605-afa7-3975a55d3712-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.517968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.517938    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"]
Oct 30 01:04:22.568471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.568234    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.572284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.572214    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.579911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.578952    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lshq\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-kube-api-access-5lshq\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.584735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.584663    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356bfb70-27c4-4904-974c-9b3230e8f643" path="/var/lib/kubelet/pods/356bfb70-27c4-4904-974c-9b3230e8f643/volumes"
Oct 30 01:04:22.591579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.590988    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9114fe-c972-4605-afa7-3975a55d3712-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:22.601025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.600989    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" event={"ID":"61231598-2fc8-4806-97a9-fc167d024f3c","Type":"ContainerStarted","Data":"ed5c6daba1e6fae60de27413366519de005609392701d5eb2c62976003a1c134"}
Oct 30 01:04:22.603551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.603522    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" event={"ID":"fd059c20-b2e8-41f5-81d8-19b831922f9e","Type":"ContainerStarted","Data":"e8fdb66e1de702f9768e608da81cdfc85eaef9a21984c72edbb16f46bd9cf029"}
Oct 30 01:04:22.605408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.605386    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" event={"ID":"fd059c20-b2e8-41f5-81d8-19b831922f9e","Type":"ContainerStarted","Data":"63c0dc60eb03e417bd58d3fa03d8d6d64f712b8198190ee3ff0774c98371d3a1"}
Oct 30 01:04:22.681907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.681478    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podStartSLOduration=1.681447086 podStartE2EDuration="1.681447086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:22.654306439 +0000 UTC m=+1304.201996945" watchObservedRunningTime="2026-10-30 01:04:22.681447086 +0000 UTC m=+1304.229137583"
Oct 30 01:04:22.846186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.846101    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:22.848496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:22.848431    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:23.286527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.278760    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"]
Oct 30 01:04:23.286527 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:23.285215    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9114fe_c972_4605_afa7_3975a55d3712.slice/crio-df7d6f4590132d934fbfaf4f91a580b1147638b03b9c4fab4e000a9d11196149 WatchSource:0}: Error finding container df7d6f4590132d934fbfaf4f91a580b1147638b03b9c4fab4e000a9d11196149: Status 404 returned error can't find the container with id df7d6f4590132d934fbfaf4f91a580b1147638b03b9c4fab4e000a9d11196149
Oct 30 01:04:23.291277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.291239    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:23.291418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.291322    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:23.291418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.291356    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:23.722478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.722389    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" event={"ID":"bd9114fe-c972-4605-afa7-3975a55d3712","Type":"ContainerStarted","Data":"0da5e3cb627ea30a958395f457ce1dbddb5266aeacfc0a7fadd0b6cb68a807bf"}
Oct 30 01:04:23.743710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.728516    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" event={"ID":"bd9114fe-c972-4605-afa7-3975a55d3712","Type":"ContainerStarted","Data":"df7d6f4590132d934fbfaf4f91a580b1147638b03b9c4fab4e000a9d11196149"}
Oct 30 01:04:23.769662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.769574    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" event={"ID":"61231598-2fc8-4806-97a9-fc167d024f3c","Type":"ContainerStarted","Data":"17e674fa19e113a3d15711509f0a22ec7d5a8de969feaf13852afc89dab52340"}
Oct 30 01:04:23.886484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.881746    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:23.886484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.881818    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:23.887704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.887669    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:04:23.887936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.887906    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 01:04:23.892243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.890581    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:04:23.892243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.890630    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:04:23.983787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:23.959593    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podStartSLOduration=1.959560462 podStartE2EDuration="1.959560462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:23.9199942 +0000 UTC m=+1305.467684697" watchObservedRunningTime="2026-10-30 01:04:23.959560462 +0000 UTC m=+1305.507250959"
Oct 30 01:04:24.052477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:24.052399    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podStartSLOduration=3.052377886 podStartE2EDuration="3.052377886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:23.982426013 +0000 UTC m=+1305.530116521" watchObservedRunningTime="2026-10-30 01:04:24.052377886 +0000 UTC m=+1305.600068383"
Oct 30 01:04:24.862378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:24.862338    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:04:24.862946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:24.862912    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:04:24.863172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:24.863125    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:04:24.863240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:24.863196    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 01:04:24.872869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:24.872834    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:04:24.873149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:24.873124    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 01:04:25.869493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:25.854332    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:04:25.869493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:25.854396    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 01:04:25.874742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:25.873878    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:04:25.874742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:25.873935    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 01:04:25.889475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:25.889426    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:04:25.889865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:25.889745    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:04:26.846518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:26.846305    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:04:26.846518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:26.846365    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:04:26.853016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:26.852977    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:04:26.853289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:26.853257    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 01:04:26.871857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:26.871738    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:04:26.871857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:26.871800    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 01:04:27.852486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:27.848888    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:04:27.852486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:27.848959    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:04:27.852486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:27.849477    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:04:27.852486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:27.849527    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 01:04:27.871813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:27.871784    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:04:27.872762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:27.872717    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 01:04:28.858518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:28.858392    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:04:28.858518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:28.858496    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 01:04:28.859335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:28.859301    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:04:28.859503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:28.859351    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:04:28.871722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:28.871690    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:04:28.871850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:28.871741    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 01:04:29.852538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:29.852063    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:04:29.852538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:29.852128    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 01:04:29.869479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:29.868333    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:04:29.869479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:29.868393    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:04:29.901481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:29.885963    5104 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:04:29.901481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:29.886022    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 01:04:30.905491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:30.899522    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:04:30.905491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:30.899594    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 01:04:31.233913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.233878    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:31.234741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.234720    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:31.235399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.235376    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:04:31.300883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.300846    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:31.301337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.301276    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:31.303753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.303725    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:04:31.951281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.951248    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:31.952281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.952250    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:31.954239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:31.954215    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:04:33.088420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:33.088356    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"]
Oct 30 01:04:33.094211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:33.094183    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e53bb6b-4wdlv"]
Oct 30 01:04:34.582827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:34.582791    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e07e027-2a84-4dd9-99b7-c21acea9c8f2" path="/var/lib/kubelet/pods/3e07e027-2a84-4dd9-99b7-c21acea9c8f2/volumes"
Oct 30 01:04:35.833146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.833111    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"]
Oct 30 01:04:35.840821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.840788    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.909906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.909865    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2be695-openshift-default-dockercfg-f5bng\""
Oct 30 01:04:35.934035 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.918249    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.934230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934127    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.934299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934232    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.934299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934269    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f6143-8983-4709-bcc6-01c5def319d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.934408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934302    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.934408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.934541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934520    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.934599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934579    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.936923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.934700    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9bp\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-kube-api-access-rb9bp\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:35.939447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:35.939419    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"]
Oct 30 01:04:36.036929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.036889    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037007    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037034    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037107    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9bp\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-kube-api-access-rb9bp\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037243    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037315    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037414    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037445    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f6143-8983-4709-bcc6-01c5def319d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037502    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037557    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.037726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037698    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.038002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.037979    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.038102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.038085    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.038361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.038332    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f6143-8983-4709-bcc6-01c5def319d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.039955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.039934    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.040047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.039954    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.056450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.056417    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9bp\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-kube-api-access-rb9bp\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.056997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.056979    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.228240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.228198    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:36.721000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.720965    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"]
Oct 30 01:04:36.724034 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:36.724006    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4f6143_8983_4709_bcc6_01c5def319d2.slice/crio-00ce952889c5c4bb6eadb28d65aa539e026e823aff5b66316eff41f6598f858e WatchSource:0}: Error finding container 00ce952889c5c4bb6eadb28d65aa539e026e823aff5b66316eff41f6598f858e: Status 404 returned error can't find the container with id 00ce952889c5c4bb6eadb28d65aa539e026e823aff5b66316eff41f6598f858e
Oct 30 01:04:36.726755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.726722    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:36.726830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.726796    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:36.726869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:36.726827    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:37.631317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:37.631281    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" event={"ID":"6f4f6143-8983-4709-bcc6-01c5def319d2","Type":"ContainerStarted","Data":"3853c548357e366cb193ff58e5e4a1074d0dadedf64d2db788c80115f146ee93"}
Oct 30 01:04:37.631760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:37.631336    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" event={"ID":"6f4f6143-8983-4709-bcc6-01c5def319d2","Type":"ContainerStarted","Data":"00ce952889c5c4bb6eadb28d65aa539e026e823aff5b66316eff41f6598f858e"}
Oct 30 01:04:38.229267 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:38.229233    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:38.235204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:38.235176    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:38.275884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:38.275828    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podStartSLOduration=3.275814081 podStartE2EDuration="3.275814081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:37.752972903 +0000 UTC m=+1319.300663399" watchObservedRunningTime="2026-10-30 01:04:38.275814081 +0000 UTC m=+1319.823504577"
Oct 30 01:04:38.640995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:38.640916    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:38.641839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:38.641817    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:04:39.108177 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:04:39.108140    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:04:40.439895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.439866    5104 scope.go:122] "RemoveContainer" containerID="d7349bd83b0233004d03bf22569d952d32e08ffa06b55a6b535690140bcfe191"
Oct 30 01:04:40.455725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.455701    5104 scope.go:122] "RemoveContainer" containerID="f5328cb72ad56eef6b4b69968ed88ddfea17130c8ec96cf8bd94f9a2948769fd"
Oct 30 01:04:40.467681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.467655    5104 scope.go:122] "RemoveContainer" containerID="ae7227ac9be50069428a4a593ca626d2d2de563955fa0cf238cbadcf1c13d851"
Oct 30 01:04:40.481446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.481418    5104 scope.go:122] "RemoveContainer" containerID="99ec928716f330092d20c3cf1fe07c2d5c66b251629e3ec1c8bdd9d74b2f8e18"
Oct 30 01:04:40.495789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.495763    5104 scope.go:122] "RemoveContainer" containerID="6c3de3a419b765e4912586a43aef50f15629dfc8798c191a544e3ec0e0b2f7ae"
Oct 30 01:04:40.509339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.509320    5104 scope.go:122] "RemoveContainer" containerID="0d645330c2c281d30a23e3b6584321e663b19e63b353770b257c43719d45ad67"
Oct 30 01:04:40.526789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.526758    5104 scope.go:122] "RemoveContainer" containerID="8b26ee47b0c6aa48eeb3e70859a90ef8f89e22968cecf1d6d019a78e2548d227"
Oct 30 01:04:40.539252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.539234    5104 scope.go:122] "RemoveContainer" containerID="53f350f0d3c39bcd7b7256d3d00e6755dd692f258534e69b51b858c4c593e98d"
Oct 30 01:04:40.550881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.550861    5104 scope.go:122] "RemoveContainer" containerID="7151ec6167b5a0b56cd11b3d1fc9bc6bbb2fc422e5673ef3f467a7a499157c0c"
Oct 30 01:04:40.560914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.560896    5104 scope.go:122] "RemoveContainer" containerID="877ed95cdc0ab85ae2a9119736af0e696595d478f1006b19ab16ef8c31046b58"
Oct 30 01:04:40.571257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.571240    5104 scope.go:122] "RemoveContainer" containerID="b672a8ef2347ca429015a5e70c5d23b71f432b02e41ed183186923d62fa73cf5"
Oct 30 01:04:40.585201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.585182    5104 scope.go:122] "RemoveContainer" containerID="08919208f41897f733481de3b620133c67a16796292288524135cb07b2855a5f"
Oct 30 01:04:40.595240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.595218    5104 scope.go:122] "RemoveContainer" containerID="62f15140599819b6d744de89704b2c8093d7e17a17c8d0ac97a808dac9b5c09c"
Oct 30 01:04:40.606933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:40.606911    5104 scope.go:122] "RemoveContainer" containerID="7c6e33216ccfa3b4af66f0baccdb2f9a26fcf1fe3d650562a9d65504bb0594f7"
Oct 30 01:04:43.954515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:43.954418    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:44.095482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:44.083699    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:04:44.095482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:44.084650    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:04:44.095482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:44.085167    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:44.372549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:44.344060    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:44.400493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:44.390263    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:44.462433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:44.462375    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:44.662540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:44.625615    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.008488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.002573    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.008488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.003665    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.130480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.108659    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:04:45.130480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.108720    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.177683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.177622    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.413690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.413572    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.446433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.419549    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.640040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.639991    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.640567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.640535    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.758480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.746080    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:45.811618 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:04:45.811576    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.098s"
Oct 30 01:04:45.811885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.811864    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"]
Oct 30 01:04:45.903514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:45.900225    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 01:04:46.066017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.013809    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:46.081975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.015631    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:04:46.083072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.051706    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"]
Oct 30 01:04:46.086803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.083410    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:04:46.087050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.087030    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:04:46.087193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.087181    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:04:46.087322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.087309    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:04:46.087486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.087472    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:04:46.087672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.087656    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:04:46.112484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.087806    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:04:46.112782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.052782    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:46.113119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.113080    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:46.113519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.113491    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:04:46.144074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.141517    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:04:46.155601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.155553    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:04:46.172680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.172642    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:04:46.196817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.196767    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:04:46.248246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.235893    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2be695-dockercfg-hrq7t\""
Oct 30 01:04:46.248246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.244850    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzxt\" (UniqueName: \"kubernetes.io/projected/4054d583-1459-402e-a08f-477bdd2e067a-kube-api-access-gnzxt\") pod \"payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:46.248246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.245107    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4054d583-1459-402e-a08f-477bdd2e067a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:46.309499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.250761    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2be695\""
Oct 30 01:04:46.377522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.349485    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzxt\" (UniqueName: \"kubernetes.io/projected/4054d583-1459-402e-a08f-477bdd2e067a-kube-api-access-gnzxt\") pod \"payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:46.377522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.349662    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4054d583-1459-402e-a08f-477bdd2e067a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:46.422554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.414702    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:46.431144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.423165    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:46.440578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.440536    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4054d583-1459-402e-a08f-477bdd2e067a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:46.450484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.449916    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:46.608365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.608278    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzxt\" (UniqueName: \"kubernetes.io/projected/4054d583-1459-402e-a08f-477bdd2e067a-kube-api-access-gnzxt\") pod \"payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:47.040041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.667476    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:47.056587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:46.763780    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:04:47.091534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.084628    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:04:47.287481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.278136    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:04:47.425537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.304170    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:04:47.425537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.324820    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:04:47.462496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.458943    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:47.488616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.488567    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:47.508486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.492861    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:47.544117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.510441    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"]
Oct 30 01:04:47.668576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.515679    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:04:47.668901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.668858    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:47.861905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.529597    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:04:47.861905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.515719    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:04:47.861905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.814813    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:47.861905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.612596    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbh6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:04:47.861905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.814893    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:47.861905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:47.694011    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:48.108764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.100644    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:04:48.223522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.202318    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"]
Oct 30 01:04:48.326007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.325956    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7jw\" (UniqueName: \"kubernetes.io/projected/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-kube-api-access-lb7jw\") pod \"payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:48.379481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.376231    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:48.477943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.477840    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:48.507488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.458110    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:48.712658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.589256    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:04:48.712658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.610942    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:04:48.731637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:48.556657    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Oct 30 01:04:49.053197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.052598    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:49.053197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.052787    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7jw\" (UniqueName: \"kubernetes.io/projected/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-kube-api-access-lb7jw\") pod \"payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:49.098861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.098809    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb" podUID="626e0a50-e9f7-4b0f-aa09-226cba9a0dfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:49.186482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.177628    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:04:49.186482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.177750    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:04:49.231512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.231444    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:04:49.360763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.360655    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9" podUID="96d0232a-84d5-44ff-b2b6-d45f390c392d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:49.495483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.482648    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7jw\" (UniqueName: \"kubernetes.io/projected/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-kube-api-access-lb7jw\") pod \"payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:49.526081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.494115    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:49.526081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.496341    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:49.526081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.496659    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 30 01:04:49.526081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.496694    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:04:49.526081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.521717    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:49.551712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.551636    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:49.701962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.701925    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:49.706668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:49.706640    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:04:50.206796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:50.177691    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:04:50.247174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:50.247141    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:04:50.364478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:50.360886    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:04:50.770004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:50.767232    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:50.770004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:50.768004    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:04:50.839540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:50.830600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:04:50.839540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:50.830994    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:04:51.296480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.291586    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:51.328736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.328698    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:51.492511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.421203    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:04:51.530897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.527541    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:04:51.647423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.647383    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" event={"ID":"4054d583-1459-402e-a08f-477bdd2e067a","Type":"ContainerStarted","Data":"224c9f3b10f42d11fb7e44584e8d6cc3fbfcfd835f72a55e485e3bb839bce600"}
Oct 30 01:04:51.722503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.686725    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:04:51.782997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.772392    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:51.789816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.788550    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:04:51.906703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:51.835897    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:04:52.154481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.154339    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:04:52.233481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.196968    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:52.299082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.299029    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:52.426349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.425576    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:04:52.460678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.460648    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:04:52.745681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.680177    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 01:04:52.745681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.692273    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:52.745681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.705850    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:04:52.795131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.795082    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:52.838089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.837555    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:52.922526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.774158    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:53.184552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:52.837926    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:04:53.251482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.250694    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:53.274490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.258725    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:53.303701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.303642    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:53.451264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.449850    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:53.556648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.556611    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:04:53.557005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.556986    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:04:53.557154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.557135    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"]
Oct 30 01:04:53.557389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.557373    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97b1dfe-l572h"]
Oct 30 01:04:53.557580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.557566    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:04:53.645303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.643644    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:04:53.654907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.654875    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:04:53.655913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.655879    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:04:53.726272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.726242    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:04:53.752737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.736715    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:04:53.755726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.755120    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:04:53.757287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.757260    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:04:53.765963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.764588    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" event={"ID":"4054d583-1459-402e-a08f-477bdd2e067a","Type":"ContainerStarted","Data":"2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b"}
Oct 30 01:04:53.765963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.764651    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:04:53.830250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:53.830210    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"]
Oct 30 01:04:54.135925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:54.135825    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"]
Oct 30 01:04:54.248900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:54.248834    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" podStartSLOduration=11.248812268 podStartE2EDuration="11.248812268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:53.922765476 +0000 UTC m=+1335.470455998" watchObservedRunningTime="2026-10-30 01:04:54.248812268 +0000 UTC m=+1335.796502765"
Oct 30 01:04:54.758712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:54.758674    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c0cace-b390-4fe2-962a-f0358df08b41" path="/var/lib/kubelet/pods/e4c0cace-b390-4fe2-962a-f0358df08b41/volumes"
Oct 30 01:04:54.806908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:54.806859    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" event={"ID":"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a","Type":"ContainerStarted","Data":"18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82"}
Oct 30 01:04:54.807208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:54.807167    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" event={"ID":"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a","Type":"ContainerStarted","Data":"32e4937b0a6fbc8a875e446b7e7c22a023ff3ff448dd62aade65b981a6d3b3db"}
Oct 30 01:04:54.813086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:54.813053    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:04:55.079574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.073125    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" podStartSLOduration=12.073100824 podStartE2EDuration="12.073100824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:54.866797607 +0000 UTC m=+1336.414488104" watchObservedRunningTime="2026-10-30 01:04:55.073100824 +0000 UTC m=+1336.620791320"
Oct 30 01:04:55.394543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.334759    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"]
Oct 30 01:04:55.584819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.584320    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:55.597945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.586778    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.625064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.625029    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-69dc90-openshift-default-dockercfg-bktbh\""
Oct 30 01:04:55.715489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.712509    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"]
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.733598    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.733792    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.733862    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-envoy\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.733917    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-data\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.733978    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-credential-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.734038    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-certs\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.734179    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5zw\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-kube-api-access-mr5zw\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.734593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.734227    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-token\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.735125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.735002    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-podinfo\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.839858    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5zw\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-kube-api-access-mr5zw\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.839922    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-token\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.839969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-podinfo\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.840036    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.840206    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.840270    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-envoy\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.840322    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-data\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.840380    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-credential-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.840440    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-certs\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.845521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.840903    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-certs\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.850491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.849289    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.850491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.849608    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-data\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.850491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.849864    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-credential-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:55.850779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:55.850532    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:56.032034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:56.029175    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-podinfo\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:56.034408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:56.032919    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5zw\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-kube-api-access-mr5zw\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:56.034408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:56.034394    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-token\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:56.064242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:56.064206    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-envoy\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:56.363516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:56.361134    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:57.414206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:57.414161    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"]
Oct 30 01:04:57.415177 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:57.415144    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23653a1_d8ae_47cf_bbf5_00ed5c3c68a7.slice/crio-d5e069a1cd6de4d4f3daa80d72af6503b8453003911017dd9b0bcd9779573efa WatchSource:0}: Error finding container d5e069a1cd6de4d4f3daa80d72af6503b8453003911017dd9b0bcd9779573efa: Status 404 returned error can't find the container with id d5e069a1cd6de4d4f3daa80d72af6503b8453003911017dd9b0bcd9779573efa
Oct 30 01:04:57.418331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:57.418292    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:57.418433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:57.418377    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:57.418433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:57.418419    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:04:58.184589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.178954    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" event={"ID":"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7","Type":"ContainerStarted","Data":"c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2"}
Oct 30 01:04:58.184589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.179031    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" event={"ID":"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7","Type":"ContainerStarted","Data":"d5e069a1cd6de4d4f3daa80d72af6503b8453003911017dd9b0bcd9779573efa"}
Oct 30 01:04:58.249660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.249597    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podStartSLOduration=3.249575623 podStartE2EDuration="3.249575623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:58.235712975 +0000 UTC m=+1339.783403472" watchObservedRunningTime="2026-10-30 01:04:58.249575623 +0000 UTC m=+1339.797266121"
Oct 30 01:04:58.304689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.304652    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"]
Oct 30 01:04:58.312504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.312478    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.327242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.326965    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb3370cb\"/\"openshift-service-ca.crt\""
Oct 30 01:04:58.327242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.327055    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb3370cb\"/\"default-dockercfg-zhntm\""
Oct 30 01:04:58.327242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.327121    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb3370cb\"/\"e2e-lifecycle-model-fb3370cb-kserve-self-signed-certs\""
Oct 30 01:04:58.327242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.327197    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb3370cb\"/\"kube-root-ca.crt\""
Oct 30 01:04:58.335612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.335584    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"]
Oct 30 01:04:58.369602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.364898    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:58.374431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.374389    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-home\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.374636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.374437    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-model-cache\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.374636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.374488    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wgm\" (UniqueName: \"kubernetes.io/projected/fea7f268-06b4-4a01-b283-f6ee7692bdf6-kube-api-access-t8wgm\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.374806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.374723    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-dshm\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.374806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.374795    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tls-certs\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.374980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.374963    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tmp-dir\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.378539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.377613    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477000    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-dshm\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477083    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tls-certs\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477255    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tmp-dir\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477397    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-home\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477423    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-model-cache\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477504    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wgm\" (UniqueName: \"kubernetes.io/projected/fea7f268-06b4-4a01-b283-f6ee7692bdf6-kube-api-access-t8wgm\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477838    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-home\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.477847    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tmp-dir\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.484917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.478129    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-model-cache\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.487710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.487681    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-dshm\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.490151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.490095    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tls-certs\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.496353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.496281    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wgm\" (UniqueName: \"kubernetes.io/projected/fea7f268-06b4-4a01-b283-f6ee7692bdf6-kube-api-access-t8wgm\") pod \"e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") " pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:58.660701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:58.657293    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:04:59.216298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:59.216225    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:59.219344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:59.218999    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:04:59.228027 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:04:59.227671    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfea7f268_06b4_4a01_b283_f6ee7692bdf6.slice/crio-a74552f6d98ea8e54becbb14df449fa09f7220489ddf086b5a0a83d1b219a4a4 WatchSource:0}: Error finding container a74552f6d98ea8e54becbb14df449fa09f7220489ddf086b5a0a83d1b219a4a4: Status 404 returned error can't find the container with id a74552f6d98ea8e54becbb14df449fa09f7220489ddf086b5a0a83d1b219a4a4
Oct 30 01:04:59.230991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:04:59.230948    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"]
Oct 30 01:05:00.235691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:00.235644    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" event={"ID":"fea7f268-06b4-4a01-b283-f6ee7692bdf6","Type":"ContainerStarted","Data":"2bdbf866d546f9a3176e444a724523f9cec792df37209bd655740cdca8f9868c"}
Oct 30 01:05:00.236106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:00.235708    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" event={"ID":"fea7f268-06b4-4a01-b283-f6ee7692bdf6","Type":"ContainerStarted","Data":"a74552f6d98ea8e54becbb14df449fa09f7220489ddf086b5a0a83d1b219a4a4"}
Oct 30 01:05:00.274368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:00.274313    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podStartSLOduration=2.274299582 podStartE2EDuration="2.274299582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:00.272710105 +0000 UTC m=+1341.820400600" watchObservedRunningTime="2026-10-30 01:05:00.274299582 +0000 UTC m=+1341.821990077"
Oct 30 01:05:00.659089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:00.659005    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:05:00.674653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:00.674607    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:05:00.674850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:00.674666    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:02.684651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:02.684608    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:05:04.644956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:02.684679    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:04.817550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:04.817517    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:05:05.154490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.153952    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"]
Oct 30 01:05:05.207878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.207838    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0eca552-cv95n"]
Oct 30 01:05:05.295722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.295687    5104 generic.go:356] "Generic (PLEG): container finished" podID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerID="818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0" exitCode=0
Oct 30 01:05:05.295921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.295761    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" event={"ID":"c081e5d8-e6c1-4739-8a2f-a363e06b8e12","Type":"ContainerDied","Data":"818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0"}
Oct 30 01:05:05.296328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.296301    5104 scope.go:122] "RemoveContainer" containerID="818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0"
Oct 30 01:05:05.298336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.298315    5104 generic.go:356] "Generic (PLEG): container finished" podID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerID="4c3ee4fad64fa2c9c765525dc9a6cd2b2355a9432c98a0af326ec7d4a9c115e8" exitCode=0
Oct 30 01:05:05.298469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.298377    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.298469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.298393    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" event={"ID":"84b90433-33b5-49ee-9cd5-2faf0655badd","Type":"ContainerDied","Data":"4c3ee4fad64fa2c9c765525dc9a6cd2b2355a9432c98a0af326ec7d4a9c115e8"}
Oct 30 01:05:05.298593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.298479    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.298593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.298530    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.298906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.298887    5104 scope.go:122] "RemoveContainer" containerID="4c3ee4fad64fa2c9c765525dc9a6cd2b2355a9432c98a0af326ec7d4a9c115e8"
Oct 30 01:05:05.300804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.300774    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.300913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.300858    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.300913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.300907    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.301188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.301163    5104 generic.go:356] "Generic (PLEG): container finished" podID="fa93bf74-fd15-47c7-8472-401b203e726e" containerID="9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371" exitCode=0
Oct 30 01:05:05.301285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.301215    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" event={"ID":"fa93bf74-fd15-47c7-8472-401b203e726e","Type":"ContainerDied","Data":"9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371"}
Oct 30 01:05:05.301682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.301660    5104 scope.go:122] "RemoveContainer" containerID="9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371"
Oct 30 01:05:05.303530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.303500    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.303634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.303581    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.303634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.303626    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:05.838132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.838093    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"]
Oct 30 01:05:05.849113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.849077    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:05.860389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.860340    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"]
Oct 30 01:05:05.865248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.865220    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-69dc90-dockercfg-n8hx9\""
Oct 30 01:05:05.865395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:05.865376    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-69dc90\""
Oct 30 01:05:06.017478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.012622    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklnk\" (UniqueName: \"kubernetes.io/projected/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-kube-api-access-bklnk\") pod \"payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:06.017478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.012929    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:06.130865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.120764    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:06.130865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.126923    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklnk\" (UniqueName: \"kubernetes.io/projected/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-kube-api-access-bklnk\") pod \"payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:06.130865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.128326    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:06.176489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.172844    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklnk\" (UniqueName: \"kubernetes.io/projected/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-kube-api-access-bklnk\") pod \"payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:06.214537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.182852    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:06.367964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.367926    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"]
Oct 30 01:05:06.470428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.470383    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b4144c2-pb877"]
Oct 30 01:05:06.572350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.572307    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"]
Oct 30 01:05:06.599657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.578651    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db01cb1-qzfjc"]
Oct 30 01:05:06.644537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.644497    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3dac020-aeba-427a-a629-6a860b6db314" path="/var/lib/kubelet/pods/d3dac020-aeba-427a-a629-6a860b6db314/volumes"
Oct 30 01:05:06.645607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.645582    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79cb0d5-8682-4ecc-9152-72505eb43b9e" path="/var/lib/kubelet/pods/f79cb0d5-8682-4ecc-9152-72505eb43b9e/volumes"
Oct 30 01:05:06.646395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.646374    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef07710-514a-4672-b81c-682651eff83f" path="/var/lib/kubelet/pods/fef07710-514a-4672-b81c-682651eff83f/volumes"
Oct 30 01:05:06.647101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.647072    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" event={"ID":"c081e5d8-e6c1-4739-8a2f-a363e06b8e12","Type":"ContainerStarted","Data":"82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f"}
Oct 30 01:05:06.647309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.647290    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" event={"ID":"84b90433-33b5-49ee-9cd5-2faf0655badd","Type":"ContainerStarted","Data":"360144cf31c29e09b299d9ab27b8628fc9edbab320fdeb488478cdee530c90dd"}
Oct 30 01:05:06.649794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:06.649766    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" event={"ID":"fa93bf74-fd15-47c7-8472-401b203e726e","Type":"ContainerStarted","Data":"aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427"}
Oct 30 01:05:07.163717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.154341    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:05:07.163717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.154502    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:07.163717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.154551    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:07.268887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.264867    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:05:07.268887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.264973    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:05:07.346483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.326270    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 01:05:07.346483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.326384    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 01:05:07.426582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.422524    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"]
Oct 30 01:05:07.541127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.541093    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:05:07.672253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.672219    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 01:05:07.672576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.672524    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 01:05:07.897482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:07.897301    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h" event={"ID":"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf","Type":"ContainerStarted","Data":"6d3bb68ae93ef79377f223e4867009b1d010b882225a393aa43c2fcfb840bd75"}
Oct 30 01:05:08.158567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.154707    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:08.158567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.154775    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:08.171316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.171280    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:05:08.273486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.271755    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 01:05:08.273486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.271825    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 01:05:08.531470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.531358    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 01:05:08.531752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.531718    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 01:05:08.662171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:08.661752    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:05:09.022629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.022582    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h" event={"ID":"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf","Type":"ContainerStarted","Data":"fd74b6126dd7351a030e70e22aefa72efcd38765b852259d744e5fbfaf6d031d"}
Oct 30 01:05:09.030774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.030744    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:09.075194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.075127    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h" podStartSLOduration=4.075105761 podStartE2EDuration="4.075105761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:09.063174948 +0000 UTC m=+1350.610865458" watchObservedRunningTime="2026-10-30 01:05:09.075105761 +0000 UTC m=+1350.622796259"
Oct 30 01:05:09.162481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.152418    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:09.162481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.152520    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:09.162481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.153146    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:05:09.256491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.250425    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 01:05:09.256491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.250536    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 01:05:09.539171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.537009    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 01:05:09.539171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.537088    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 01:05:09.605481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.596672    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:05:09.618485 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:05:09.607765    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:05:09.860535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.805277    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:09.940532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.841487    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:05:10.072541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.841528    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:05:10.072541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.853338    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:10.116074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:09.861692    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:05:10.129271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.129230    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:05:10.153798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.151502    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:10.153798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.153712    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:10.263479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.259594    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 01:05:10.263479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.259656    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 01:05:10.493952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.493904    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:10.515196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.515157    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:10.544753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.544071    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 01:05:10.544753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.544144    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 01:05:10.747536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.746743    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:05:10.961481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.926446    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:10.961481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:10.927069    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:05:11.261182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.261135    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:05:11.262515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.262479    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:05:11.263278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.263255    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:05:11.264953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.264926    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:11.265698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.265670    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:05:11.266365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.266342    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:05:11.266565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.266550    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:05:11.268800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.268778    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 01:05:11.270097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.270072    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 01:05:11.345488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.276100    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Oct 30 01:05:11.345488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.304233    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": context deadline exceeded"
Oct 30 01:05:11.345488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.304972    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:11.345488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.305010    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:11.523534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.522753    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:05:11.523534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.523376    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:05:11.619825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.619700    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 01:05:11.620291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.620168    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 01:05:11.770970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.770922    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:11.771351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.771276    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:11.777247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.777022    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:05:11.939497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:11.929720    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:12.008042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.008010    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:05:12.266156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.266101    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:12.289491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.286988    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:05:12.339942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.339894    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:12.360107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.360060    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:12.360704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.360500    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:05:12.366316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.366291    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:05:12.393476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.393435    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:05:12.393879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.393860    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:05:12.453082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.420605    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 01:05:12.453373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.453341    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 01:05:12.528749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.528633    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:12.535435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.535405    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:05:12.545496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.538831    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 01:05:12.546106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.546067    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 01:05:12.770481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.681300    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:12.770481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.700441    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:12.770481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.700516    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:12.796734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:12.780944    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:13.154057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.153957    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:13.156158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.156027    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:05:13.156551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.156537    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:05:13.166811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.166186    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:13.172870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.172832    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:13.196015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.195986    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:05:13.196315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.196300    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:05:13.311768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.304645    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:05:13.311768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.281817    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 01:05:13.311768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.305828    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 01:05:13.311768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.290531    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:05:13.311768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.307392    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:05:13.421629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.421587    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:05:13.525483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.522811    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:05:13.669488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.668314    5104 patch_prober.go:29] interesting pod/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 01:05:13.669488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.668375    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 01:05:13.705887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.705797    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:05:13.834757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.834294    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:05:13.890725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:13.890691    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:05:14.147631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.147515    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:14.147859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.147753    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:14.242903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.240878    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:05:14.242903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.241358    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:05:14.242903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.242706    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:05:14.758481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.741801    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:05:14.759447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.759422    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:05:14.810477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:14.805847    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:05:15.152019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:15.151923    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 01:05:15.152279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:15.152249    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 01:05:16.151431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:16.151397    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:05:16.152811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:16.152786    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:05:21.472597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:21.472565    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:05:23.072598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:23.072541    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"]
Oct 30 01:05:23.088485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:23.088414    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba58e55-fxrdq"]
Oct 30 01:05:24.593962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:24.593920    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c423303f-001a-43c2-96bf-2d5377abe4ef" path="/var/lib/kubelet/pods/c423303f-001a-43c2-96bf-2d5377abe4ef/volumes"
Oct 30 01:05:25.028778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:25.028736    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"]
Oct 30 01:05:25.040850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:25.040819    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7bb922-77d5q"]
Oct 30 01:05:26.602005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:26.601958    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e0f99b-241d-4128-a42a-5a5dc24ea4fa" path="/var/lib/kubelet/pods/84e0f99b-241d-4128-a42a-5a5dc24ea4fa/volumes"
Oct 30 01:05:30.028805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:30.028758    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"]
Oct 30 01:05:30.040244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:30.040207    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-756ff380-xrdhb"]
Oct 30 01:05:30.579224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:30.579188    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6187ebe3-e4f8-401f-93a8-e9d60226b8f7" path="/var/lib/kubelet/pods/6187ebe3-e4f8-401f-93a8-e9d60226b8f7/volumes"
Oct 30 01:05:35.088479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:35.077765    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"]
Oct 30 01:05:35.119522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:35.119482    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"]
Oct 30 01:05:35.132067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:35.131994    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeacf087-4g2vl"]
Oct 30 01:05:35.147301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:35.147248    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88485e4-cn6tk"]
Oct 30 01:05:36.027387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:36.027351    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"]
Oct 30 01:05:36.037784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:36.037750    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf56dca-64qhl"]
Oct 30 01:05:36.580797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:36.580762    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e59b88-b95e-4cc5-8507-63901accefb1" path="/var/lib/kubelet/pods/52e59b88-b95e-4cc5-8507-63901accefb1/volumes"
Oct 30 01:05:36.581317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:36.581303    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8" path="/var/lib/kubelet/pods/9dd5844e-6bb5-46f9-84a1-6aa20b91ecc8/volumes"
Oct 30 01:05:36.581866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:36.581851    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb64421-908c-43f1-b1d5-cb81761f5998" path="/var/lib/kubelet/pods/fbb64421-908c-43f1-b1d5-cb81761f5998/volumes"
Oct 30 01:05:37.760304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.760263    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"]
Oct 30 01:05:37.767535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.767508    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.820280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.820241    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-260268-openshift-default-dockercfg-smvnw\""
Oct 30 01:05:37.830531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.830500    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"]
Oct 30 01:05:37.845761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.845720    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-data\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.845844    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-podinfo\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.845926    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-token\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.845972    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-envoy\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.846019    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-certs\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.846181    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-socket\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.846234    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsjqm\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-kube-api-access-hsjqm\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.846277    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76516bd-ba32-46c7-8a0f-878527bc1633-istiod-ca-cert\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.846774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.846753    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-credential-socket\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.947956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.947919    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-credential-socket\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.947966    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-data\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948022    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-podinfo\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948043    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-token\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948071    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-envoy\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948111    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-certs\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948273    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-socket\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948330    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsjqm\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-kube-api-access-hsjqm\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948372    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76516bd-ba32-46c7-8a0f-878527bc1633-istiod-ca-cert\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948442    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-credential-socket\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948480    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-data\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948574    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-certs\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.948752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.948654    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-socket\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.949067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.949049    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76516bd-ba32-46c7-8a0f-878527bc1633-istiod-ca-cert\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.950506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.950447    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-envoy\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:37.950628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:37.950607    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-podinfo\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:38.041525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:38.028232    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsjqm\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-kube-api-access-hsjqm\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:38.057616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:38.029686    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-token\") pod \"e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") " pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:38.128919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:38.128885    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:38.910542 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:05:38.910510    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76516bd_ba32_46c7_8a0f_878527bc1633.slice/crio-b60a46fc27aed9ae44f59c87e1e17cb67e928649fe13f7bb983aef57a8c065b0 WatchSource:0}: Error finding container b60a46fc27aed9ae44f59c87e1e17cb67e928649fe13f7bb983aef57a8c065b0: Status 404 returned error can't find the container with id b60a46fc27aed9ae44f59c87e1e17cb67e928649fe13f7bb983aef57a8c065b0
Oct 30 01:05:38.913095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:38.913060    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"]
Oct 30 01:05:38.913626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:38.913593    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:38.913720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:38.913679    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:38.913720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:38.913707    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:39.114339 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:05:39.114267    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:05:39.740997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:39.740956    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" event={"ID":"a76516bd-ba32-46c7-8a0f-878527bc1633","Type":"ContainerStarted","Data":"8b913aab1f84d28d56ead6e21e01913b007640830de5fb1aa93f591200d44ace"}
Oct 30 01:05:39.741197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:39.741036    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" event={"ID":"a76516bd-ba32-46c7-8a0f-878527bc1633","Type":"ContainerStarted","Data":"b60a46fc27aed9ae44f59c87e1e17cb67e928649fe13f7bb983aef57a8c065b0"}
Oct 30 01:05:39.797070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:39.797006    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podStartSLOduration=2.796989259 podStartE2EDuration="2.796989259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:39.791001012 +0000 UTC m=+1381.338691507" watchObservedRunningTime="2026-10-30 01:05:39.796989259 +0000 UTC m=+1381.344679756"
Oct 30 01:05:40.140588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:40.135229    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:40.144093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:40.144066    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:40.813529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:40.808734    5104 scope.go:122] "RemoveContainer" containerID="4b40bde495243869c7b7cc5351bdbad37975ebaeea1116a4b541c160e8b7f20c"
Oct 30 01:05:40.901489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:40.881331    5104 scope.go:122] "RemoveContainer" containerID="75b012db3d60c0b16d881ed661e099fcc91647b1fc1821bdd932ffc765c57cf6"
Oct 30 01:05:40.984491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:40.982104    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:40.984491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:40.984293    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:05:41.480809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:41.480763    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:05:44.115641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.115608    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"]
Oct 30 01:05:44.123481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.123427    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.123670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.123577    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"]
Oct 30 01:05:44.195479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.161958    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-260268\""
Oct 30 01:05:44.216508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.197183    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-260268-dockercfg-hjrpg\""
Oct 30 01:05:44.231581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.219276    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twmr\" (UniqueName: \"kubernetes.io/projected/0083837f-171e-42ef-a693-15f8952f21b7-kube-api-access-8twmr\") pod \"payload-processing-e2e-api-a-260268-5558bb6674-bwd6f\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.231581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.219371    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0083837f-171e-42ef-a693-15f8952f21b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-260268-5558bb6674-bwd6f\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.370531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.339689    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"]
Oct 30 01:05:44.376694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.376663    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:44.441484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.419268    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twmr\" (UniqueName: \"kubernetes.io/projected/0083837f-171e-42ef-a693-15f8952f21b7-kube-api-access-8twmr\") pod \"payload-processing-e2e-api-a-260268-5558bb6674-bwd6f\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.441484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.419381    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0083837f-171e-42ef-a693-15f8952f21b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-260268-5558bb6674-bwd6f\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.441484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.420685    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0083837f-171e-42ef-a693-15f8952f21b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-260268-5558bb6674-bwd6f\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.643560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.630814    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af0b59c-9cfa-4255-b94c-6473ae258b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:44.643560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.631324    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9qj\" (UniqueName: \"kubernetes.io/projected/2af0b59c-9cfa-4255-b94c-6473ae258b48-kube-api-access-dw9qj\") pod \"payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:44.757812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.757777    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twmr\" (UniqueName: \"kubernetes.io/projected/0083837f-171e-42ef-a693-15f8952f21b7-kube-api-access-8twmr\") pod \"payload-processing-e2e-api-a-260268-5558bb6674-bwd6f\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.781843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.773210    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9qj\" (UniqueName: \"kubernetes.io/projected/2af0b59c-9cfa-4255-b94c-6473ae258b48-kube-api-access-dw9qj\") pod \"payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:44.802480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.800984    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af0b59c-9cfa-4255-b94c-6473ae258b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:44.809478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.802984    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:44.818356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.818300    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af0b59c-9cfa-4255-b94c-6473ae258b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:44.832505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.832418    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"]
Oct 30 01:05:44.847592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:44.847314    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9qj\" (UniqueName: \"kubernetes.io/projected/2af0b59c-9cfa-4255-b94c-6473ae258b48-kube-api-access-dw9qj\") pod \"payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:45.132437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:45.132399    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:45.224819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:45.224780    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"]
Oct 30 01:05:45.225043 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:05:45.225020    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0083837f_171e_42ef_a693_15f8952f21b7.slice/crio-4f64e19dba3d3e2fdd46318cd0cea7072995b961b30a1bce64b6c44b14e1e607 WatchSource:0}: Error finding container 4f64e19dba3d3e2fdd46318cd0cea7072995b961b30a1bce64b6c44b14e1e607: Status 404 returned error can't find the container with id 4f64e19dba3d3e2fdd46318cd0cea7072995b961b30a1bce64b6c44b14e1e607
Oct 30 01:05:45.799478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:45.797895    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"]
Oct 30 01:05:45.853555 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:05:45.838431    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af0b59c_9cfa_4255_b94c_6473ae258b48.slice/crio-b167d013f7ba7a115ae436d5129e75406789b1a2add4cc8c7bf37fee377f030b WatchSource:0}: Error finding container b167d013f7ba7a115ae436d5129e75406789b1a2add4cc8c7bf37fee377f030b: Status 404 returned error can't find the container with id b167d013f7ba7a115ae436d5129e75406789b1a2add4cc8c7bf37fee377f030b
Oct 30 01:05:46.147513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.147370    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z" event={"ID":"2af0b59c-9cfa-4255-b94c-6473ae258b48","Type":"ContainerStarted","Data":"1dd9531e5ce18415f345f7e0f0e9d5619e2ca536e85efb75833dd55658d6b767"}
Oct 30 01:05:46.147513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.147440    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z" event={"ID":"2af0b59c-9cfa-4255-b94c-6473ae258b48","Type":"ContainerStarted","Data":"b167d013f7ba7a115ae436d5129e75406789b1a2add4cc8c7bf37fee377f030b"}
Oct 30 01:05:46.147513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.147507    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:46.149687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.149654    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f" event={"ID":"0083837f-171e-42ef-a693-15f8952f21b7","Type":"ContainerStarted","Data":"3d1d9c849f694106fc01143a124bc2b201e3559ebb6520d561cbd27548ecb312"}
Oct 30 01:05:46.149845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.149709    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f" event={"ID":"0083837f-171e-42ef-a693-15f8952f21b7","Type":"ContainerStarted","Data":"4f64e19dba3d3e2fdd46318cd0cea7072995b961b30a1bce64b6c44b14e1e607"}
Oct 30 01:05:46.149845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.149736    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:46.218592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.218138    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z" podStartSLOduration=2.218116152 podStartE2EDuration="2.218116152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:46.189720028 +0000 UTC m=+1387.737410524" watchObservedRunningTime="2026-10-30 01:05:46.218116152 +0000 UTC m=+1387.765806649"
Oct 30 01:05:46.241216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:46.241144    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f" podStartSLOduration=2.241125816 podStartE2EDuration="2.241125816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:46.233379409 +0000 UTC m=+1387.781069905" watchObservedRunningTime="2026-10-30 01:05:46.241125816 +0000 UTC m=+1387.788816313"
Oct 30 01:05:50.031865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:50.031779    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"]
Oct 30 01:05:50.042768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:50.042737    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-506b3861-j25df"]
Oct 30 01:05:50.789482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:50.779722    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c962e72-cff0-4575-b4b5-beabdfecef5c" path="/var/lib/kubelet/pods/6c962e72-cff0-4575-b4b5-beabdfecef5c/volumes"
Oct 30 01:05:51.027811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:51.027769    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"]
Oct 30 01:05:51.038861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:51.038829    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c751398f-spq5f"]
Oct 30 01:05:52.032682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:52.032645    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"]
Oct 30 01:05:52.043776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:52.043743    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea7cd55-zvwpm"]
Oct 30 01:05:52.644806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:52.644759    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1591081c-a77d-455b-9ba2-b01dc213fa47" path="/var/lib/kubelet/pods/1591081c-a77d-455b-9ba2-b01dc213fa47/volumes"
Oct 30 01:05:52.645749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:52.645727    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52c6cdc-e6de-4f33-871f-52e4b00e6494" path="/var/lib/kubelet/pods/a52c6cdc-e6de-4f33-871f-52e4b00e6494/volumes"
Oct 30 01:05:53.122506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.095529    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"]
Oct 30 01:05:53.144548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.141615    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.154688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.154642    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"]
Oct 30 01:05:53.159043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.158862    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:05:53.159732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.159686    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:05:53.239695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.239664    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:05:53.239845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.239717    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:05:53.257277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.257237    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.257798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.257771    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.257947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.257919    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w57v\" (UniqueName: \"kubernetes.io/projected/6b033b4a-f168-4a05-ab18-4c605db015e0-kube-api-access-5w57v\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.282158 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:05:53.282119    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@8-25548-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23653a1_d8ae_47cf_bbf5_00ed5c3c68a7.slice/crio-c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:05:53.329593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.329533    5104 generic.go:356] "Generic (PLEG): container finished" podID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerID="c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2" exitCode=0
Oct 30 01:05:53.329884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.329856    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" event={"ID":"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7","Type":"ContainerDied","Data":"c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2"}
Oct 30 01:05:53.331513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.330496    5104 scope.go:122] "RemoveContainer" containerID="c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2"
Oct 30 01:05:53.334172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.334132    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:53.334298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.334227    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:53.334298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.334281    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:05:53.367255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.367222    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.367417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.367328    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w57v\" (UniqueName: \"kubernetes.io/projected/6b033b4a-f168-4a05-ab18-4c605db015e0-kube-api-access-5w57v\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.367417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.367386    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.368961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.368934    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.398130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.397354    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w57v\" (UniqueName: \"kubernetes.io/projected/6b033b4a-f168-4a05-ab18-4c605db015e0-kube-api-access-5w57v\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.420695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.420665    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:53.473680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:53.472280    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:54.220757 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:05:54.220727    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b033b4a_f168_4a05_ab18_4c605db015e0.slice/crio-4dcbdd3f7e544c5857070edea800cda5241b5749b2428cfa406618351c0ae356 WatchSource:0}: Error finding container 4dcbdd3f7e544c5857070edea800cda5241b5749b2428cfa406618351c0ae356: Status 404 returned error can't find the container with id 4dcbdd3f7e544c5857070edea800cda5241b5749b2428cfa406618351c0ae356
Oct 30 01:05:54.223966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:54.223941    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"]
Oct 30 01:05:54.355512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:54.355442    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" event={"ID":"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7","Type":"ContainerStarted","Data":"9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6"}
Oct 30 01:05:54.357250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:54.357220    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn" event={"ID":"6b033b4a-f168-4a05-ab18-4c605db015e0","Type":"ContainerStarted","Data":"4dcbdd3f7e544c5857070edea800cda5241b5749b2428cfa406618351c0ae356"}
Oct 30 01:05:54.364468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:54.364432    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:05:54.364669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:54.364648    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:05:54.364733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:54.364696    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:05:55.364297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:55.364264    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:05:55.364814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:55.364317    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:05:55.370300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:55.370272    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn" event={"ID":"6b033b4a-f168-4a05-ab18-4c605db015e0","Type":"ContainerStarted","Data":"942f44cb56d8954e1dbdac8d57c3bdf69e2068786d230b6d9aaefaaf671e32f4"}
Oct 30 01:05:55.412129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:55.412078    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn" podStartSLOduration=3.412061649 podStartE2EDuration="3.412061649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:55.410340329 +0000 UTC m=+1396.958030825" watchObservedRunningTime="2026-10-30 01:05:55.412061649 +0000 UTC m=+1396.959752164"
Oct 30 01:05:56.364620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:56.364586    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:05:56.365051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:56.364645    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:05:56.434533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:56.429126    5104 generic.go:356] "Generic (PLEG): container finished" podID="6b033b4a-f168-4a05-ab18-4c605db015e0" containerID="942f44cb56d8954e1dbdac8d57c3bdf69e2068786d230b6d9aaefaaf671e32f4" exitCode=0
Oct 30 01:05:56.434533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:56.429373    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn" event={"ID":"6b033b4a-f168-4a05-ab18-4c605db015e0","Type":"ContainerDied","Data":"942f44cb56d8954e1dbdac8d57c3bdf69e2068786d230b6d9aaefaaf671e32f4"}
Oct 30 01:05:57.064477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.063507    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"]
Oct 30 01:05:57.071540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.070794    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760dfd7e-d59p4"]
Oct 30 01:05:57.162271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.162238    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:05:57.162752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.162730    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:05:57.370548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.370394    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:05:57.370548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.370512    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:05:57.704483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.698696    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:57.758148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.758118    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-xks-ca\") pod \"6b033b4a-f168-4a05-ab18-4c605db015e0\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") "
Oct 30 01:05:57.758340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.758193    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b033b4a-f168-4a05-ab18-4c605db015e0-kube-api-access-5w57v\" (UniqueName: \"kubernetes.io/projected/6b033b4a-f168-4a05-ab18-4c605db015e0-kube-api-access-5w57v\") pod \"6b033b4a-f168-4a05-ab18-4c605db015e0\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") "
Oct 30 01:05:57.758340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.758231    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-openshift-ca\") pod \"6b033b4a-f168-4a05-ab18-4c605db015e0\" (UID: \"6b033b4a-f168-4a05-ab18-4c605db015e0\") "
Oct 30 01:05:57.758767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.758732    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-openshift-ca" pod "6b033b4a-f168-4a05-ab18-4c605db015e0" (UID: "6b033b4a-f168-4a05-ab18-4c605db015e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:57.759248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.759227    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:05:57.760276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.760253    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-xks-ca" pod "6b033b4a-f168-4a05-ab18-4c605db015e0" (UID: "6b033b4a-f168-4a05-ab18-4c605db015e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:57.760441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.760420    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b033b4a-f168-4a05-ab18-4c605db015e0-kube-api-access-5w57v" pod "6b033b4a-f168-4a05-ab18-4c605db015e0" (UID: "6b033b4a-f168-4a05-ab18-4c605db015e0"). InnerVolumeSpecName "kube-api-access-5w57v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:57.861367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.861330    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b033b4a-f168-4a05-ab18-4c605db015e0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:05:57.861367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.861368    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w57v\" (UniqueName: \"kubernetes.io/projected/6b033b4a-f168-4a05-ab18-4c605db015e0-kube-api-access-5w57v\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:05:57.912641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.912556    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"]
Oct 30 01:05:57.915658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.915586    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6b033b4a-f168-4a05-ab18-4c605db015e0" containerName="revoke-keys"
Oct 30 01:05:57.915658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.915621    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b033b4a-f168-4a05-ab18-4c605db015e0" containerName="revoke-keys"
Oct 30 01:05:57.915892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.915867    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b033b4a-f168-4a05-ab18-4c605db015e0" containerName="revoke-keys"
Oct 30 01:05:57.921167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.921140    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"]
Oct 30 01:05:57.921508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.921236    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:57.964424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.964379    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:57.964979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.964953    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfkr\" (UniqueName: \"kubernetes.io/projected/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-kube-api-access-cgfkr\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:57.965104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:57.965065    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.066528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.066479    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.066701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.066556    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfkr\" (UniqueName: \"kubernetes.io/projected/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-kube-api-access-cgfkr\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.066701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.066649    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.067333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.067310    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.069066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.069043    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.105988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.105959    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfkr\" (UniqueName: \"kubernetes.io/projected/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-kube-api-access-cgfkr\") pod \"maas-api-revoke-sub-a087f059-dsbwm\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.243303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.243268    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:05:58.369483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.365618    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:05:58.369483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.365681    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:05:58.564975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.564942    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn" event={"ID":"6b033b4a-f168-4a05-ab18-4c605db015e0","Type":"ContainerDied","Data":"4dcbdd3f7e544c5857070edea800cda5241b5749b2428cfa406618351c0ae356"}
Oct 30 01:05:58.565491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.565450    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcbdd3f7e544c5857070edea800cda5241b5749b2428cfa406618351c0ae356"
Oct 30 01:05:58.565666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.565652    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"
Oct 30 01:05:58.645396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.645358    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd359f6a-f6c5-48f2-a822-5221332bb572" path="/var/lib/kubelet/pods/cd359f6a-f6c5-48f2-a822-5221332bb572/volumes"
Oct 30 01:05:58.955695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:58.955663    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"]
Oct 30 01:05:59.027698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.027657    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"]
Oct 30 01:05:59.059512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.050039    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"]
Oct 30 01:05:59.059512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.050172    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.137481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.077853    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmhf\" (UniqueName: \"kubernetes.io/projected/093e06f8-d249-4474-857d-4818e3df198d-kube-api-access-2bmhf\") pod \"payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.163484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.138147    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093e06f8-d249-4474-857d-4818e3df198d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.171531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.107617    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-288da6-dockercfg-ndb5d\""
Oct 30 01:05:59.200556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.106779    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-288da6\""
Oct 30 01:05:59.229746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.217029    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:05:59.308002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.274040    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093e06f8-d249-4474-857d-4818e3df198d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.308002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.274254    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmhf\" (UniqueName: \"kubernetes.io/projected/093e06f8-d249-4474-857d-4818e3df198d-kube-api-access-2bmhf\") pod \"payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.435833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.435171    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:05:59.435833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.435239    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:05:59.601117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.601081    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093e06f8-d249-4474-857d-4818e3df198d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.644612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.644579    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmhf\" (UniqueName: \"kubernetes.io/projected/093e06f8-d249-4474-857d-4818e3df198d-kube-api-access-2bmhf\") pod \"payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.676324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.676291    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:05:59.721712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:05:59.721667    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm" event={"ID":"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3","Type":"ContainerStarted","Data":"6ed3508102c8f27ac6b78b11fb1d44a1953a6ee743bf78a1e467c427a2027e55"}
Oct 30 01:06:00.029103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.029014    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"]
Oct 30 01:06:00.035989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.035958    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbddc36-ss8zw"]
Oct 30 01:06:00.059842 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:06:00.059811    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093e06f8_d249_4474_857d_4818e3df198d.slice/crio-d232b137c6b66dbfa1b7e49e7d7a95d4fe16d217c98e027f553b2f510c1542ad WatchSource:0}: Error finding container d232b137c6b66dbfa1b7e49e7d7a95d4fe16d217c98e027f553b2f510c1542ad: Status 404 returned error can't find the container with id d232b137c6b66dbfa1b7e49e7d7a95d4fe16d217c98e027f553b2f510c1542ad
Oct 30 01:06:00.061137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.061107    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"]
Oct 30 01:06:00.365537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.365509    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:06:00.365720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.365570    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:06:00.645066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.644946    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0e75a3-027a-41f2-9e15-905c617efcd1" path="/var/lib/kubelet/pods/df0e75a3-027a-41f2-9e15-905c617efcd1/volumes"
Oct 30 01:06:00.745850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.745803    5104 generic.go:356] "Generic (PLEG): container finished" podID="bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" containerID="423a6475ab8052271a5c3bda0d19a1b7577567fd6cba8faf9ffae55ef234fad3" exitCode=0
Oct 30 01:06:00.746055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.745963    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm" event={"ID":"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3","Type":"ContainerDied","Data":"423a6475ab8052271a5c3bda0d19a1b7577567fd6cba8faf9ffae55ef234fad3"}
Oct 30 01:06:00.748041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.748014    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79" event={"ID":"093e06f8-d249-4474-857d-4818e3df198d","Type":"ContainerStarted","Data":"69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f"}
Oct 30 01:06:00.748191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.748063    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79" event={"ID":"093e06f8-d249-4474-857d-4818e3df198d","Type":"ContainerStarted","Data":"d232b137c6b66dbfa1b7e49e7d7a95d4fe16d217c98e027f553b2f510c1542ad"}
Oct 30 01:06:00.748191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.748106    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:06:00.825677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:00.825625    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79" podStartSLOduration=2.825609344 podStartE2EDuration="2.825609344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:00.820579784 +0000 UTC m=+1402.368270280" watchObservedRunningTime="2026-10-30 01:06:00.825609344 +0000 UTC m=+1402.373299891"
Oct 30 01:06:01.369753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:01.369722    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:06:01.371173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:01.371149    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:06:02.123846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.123816    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:06:02.210498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.210429    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-kube-api-access-cgfkr\" (UniqueName: \"kubernetes.io/projected/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-kube-api-access-cgfkr\") pod \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") "
Oct 30 01:06:02.210712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.210689    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-xks-ca\") pod \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") "
Oct 30 01:06:02.210962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.210914    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-openshift-ca\") pod \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\" (UID: \"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3\") "
Oct 30 01:06:02.211513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.211433    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-openshift-ca" pod "bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" (UID: "bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:02.211734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.211716    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.220037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.219983    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-kube-api-access-cgfkr" pod "bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" (UID: "bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3"). InnerVolumeSpecName "kube-api-access-cgfkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:02.221845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.221804    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-xks-ca" pod "bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" (UID: "bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:02.313656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.313568    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfkr\" (UniqueName: \"kubernetes.io/projected/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-kube-api-access-cgfkr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.313656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.313601    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.778323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.778287    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"
Oct 30 01:06:02.778490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.778286    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm" event={"ID":"bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3","Type":"ContainerDied","Data":"6ed3508102c8f27ac6b78b11fb1d44a1953a6ee743bf78a1e467c427a2027e55"}
Oct 30 01:06:02.778490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:02.778427    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed3508102c8f27ac6b78b11fb1d44a1953a6ee743bf78a1e467c427a2027e55"
Oct 30 01:06:03.452135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:03.452100    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"]
Oct 30 01:06:04.028221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:04.028177    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"]
Oct 30 01:06:04.033643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:04.033613    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408e466-7mkx8"]
Oct 30 01:06:04.583069 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:04.583030    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e707d38-566a-4080-ad70-7c186ced7bd5" path="/var/lib/kubelet/pods/6e707d38-566a-4080-ad70-7c186ced7bd5/volumes"
Oct 30 01:06:08.452806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:08.452742    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerName="payload-pre-processing" containerID="cri-o://1faa4c2ff1e6f40e166761025c923be7b751062118b9cedb6afa2d9b5376ccca" gracePeriod=25
Oct 30 01:06:09.042487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:09.042431    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:06:09.065637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:09.065587    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"]
Oct 30 01:06:09.070888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:09.070858    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba301f8-dj6q8"]
Oct 30 01:06:09.107298 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:06:09.107258    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:06:09.312561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:09.312432    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:06:10.146978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.146938    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"]
Oct 30 01:06:10.148139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.148112    5104 cpu_manager.go:414] "containerMap: removing container" podUID="bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" containerName="revoke-keys"
Oct 30 01:06:10.148139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.148134    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" containerName="revoke-keys"
Oct 30 01:06:10.148352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.148266    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" containerName="revoke-keys"
Oct 30 01:06:10.153982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.153953    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.187713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.187685    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a3204e-openshift-default-dockercfg-mlg67\""
Oct 30 01:06:10.188782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.188757    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"]
Oct 30 01:06:10.254714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.254671    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psk4k\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-kube-api-access-psk4k\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.254892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.254812    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-data\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.254892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.254845    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42f8ae0-21a7-446c-95f0-06b568a26d87-istiod-ca-cert\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.254969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.254897    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-certs\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.255005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.254983    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-token\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.255047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.255036    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-socket\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.255209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.255189    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-credential-socket\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.255249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.255238    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-envoy\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.255400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.255386    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-podinfo\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371073    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-podinfo\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371134    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psk4k\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-kube-api-access-psk4k\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371216    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-data\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371242    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42f8ae0-21a7-446c-95f0-06b568a26d87-istiod-ca-cert\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371309    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-certs\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371367    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-token\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371445    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-socket\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371636    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-credential-socket\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.371711    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-envoy\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.372984    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-certs\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.379853    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-data\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.380111    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-credential-socket\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.382914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.382365    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-socket\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.384548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.384523    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42f8ae0-21a7-446c-95f0-06b568a26d87-istiod-ca-cert\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.399688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.392251    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-envoy\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.407824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.406092    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-podinfo\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.440484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.420374    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psk4k\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-kube-api-access-psk4k\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.440484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.428677    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-token\") pod \"e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") " pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.499484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.493984    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:10.791770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:10.791732    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a155edeb-19fa-4020-ac91-c7a0a17bab59" path="/var/lib/kubelet/pods/a155edeb-19fa-4020-ac91-c7a0a17bab59/volumes"
Oct 30 01:06:11.347506 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:06:11.347470    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42f8ae0_21a7_446c_95f0_06b568a26d87.slice/crio-465bd0daa5f4cc66082a9dfa83da754654b5602bac77271512b8cc7fd9e2ab7c WatchSource:0}: Error finding container 465bd0daa5f4cc66082a9dfa83da754654b5602bac77271512b8cc7fd9e2ab7c: Status 404 returned error can't find the container with id 465bd0daa5f4cc66082a9dfa83da754654b5602bac77271512b8cc7fd9e2ab7c
Oct 30 01:06:11.348305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:11.348277    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"]
Oct 30 01:06:11.350573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:11.350526    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:11.350681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:11.350606    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:11.350681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:11.350657    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:11.768665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:11.768637    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:06:12.327640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:12.327467    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" event={"ID":"d42f8ae0-21a7-446c-95f0-06b568a26d87","Type":"ContainerStarted","Data":"ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39"}
Oct 30 01:06:12.327640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:12.327574    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" event={"ID":"d42f8ae0-21a7-446c-95f0-06b568a26d87","Type":"ContainerStarted","Data":"465bd0daa5f4cc66082a9dfa83da754654b5602bac77271512b8cc7fd9e2ab7c"}
Oct 30 01:06:12.402058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:12.402005    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podStartSLOduration=2.401988994 podStartE2EDuration="2.401988994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:12.385804139 +0000 UTC m=+1413.933494676" watchObservedRunningTime="2026-10-30 01:06:12.401988994 +0000 UTC m=+1413.949679550"
Oct 30 01:06:12.497247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:12.497212    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:12.638533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:12.626658    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:13.340567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:13.340534    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:13.341649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:13.341625    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:06:16.051355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.051313    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"]
Oct 30 01:06:16.104572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.104535    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"]
Oct 30 01:06:16.122074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.122000    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03035d5-2bhns"]
Oct 30 01:06:16.132101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.132040    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c40345-j4578"]
Oct 30 01:06:16.579786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.579747    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22657d84-b586-4fc8-8cb5-b9ab23415556" path="/var/lib/kubelet/pods/22657d84-b586-4fc8-8cb5-b9ab23415556/volumes"
Oct 30 01:06:16.580257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.580241    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2d759f-e830-4661-bb7b-be778a34f031" path="/var/lib/kubelet/pods/db2d759f-e830-4661-bb7b-be778a34f031/volumes"
Oct 30 01:06:16.649869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.649839    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"]
Oct 30 01:06:16.656280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.656249    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:16.672017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.671989    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a3204e\""
Oct 30 01:06:16.674650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.674621    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"]
Oct 30 01:06:16.679152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.679130    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a3204e-dockercfg-kv4xr\""
Oct 30 01:06:16.973592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.845515    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"]
Oct 30 01:06:16.973592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.857368    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f627cdcb-aed4-496b-9137-2a8a60c48827-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:16.973592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:16.857519    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4c78\" (UniqueName: \"kubernetes.io/projected/f627cdcb-aed4-496b-9137-2a8a60c48827-kube-api-access-l4c78\") pod \"payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:17.009481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.003079    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f627cdcb-aed4-496b-9137-2a8a60c48827-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:17.009481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.003201    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4c78\" (UniqueName: \"kubernetes.io/projected/f627cdcb-aed4-496b-9137-2a8a60c48827-kube-api-access-l4c78\") pod \"payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:17.009481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.004647    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f627cdcb-aed4-496b-9137-2a8a60c48827-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:17.059830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.047739    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.137548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.112181    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4c78\" (UniqueName: \"kubernetes.io/projected/f627cdcb-aed4-496b-9137-2a8a60c48827-kube-api-access-l4c78\") pod \"payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:17.139159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.136559    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfc5s\" (UniqueName: \"kubernetes.io/projected/5775e7cb-9494-47be-b959-4a0cc2a63356-kube-api-access-sfc5s\") pod \"payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.139526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.139503    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775e7cb-9494-47be-b959-4a0cc2a63356-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.155478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.154447    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"]
Oct 30 01:06:17.225124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.217571    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"]
Oct 30 01:06:17.245529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.245492    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfc5s\" (UniqueName: \"kubernetes.io/projected/5775e7cb-9494-47be-b959-4a0cc2a63356-kube-api-access-sfc5s\") pod \"payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.246008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.245976    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775e7cb-9494-47be-b959-4a0cc2a63356-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.248183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.248157    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775e7cb-9494-47be-b959-4a0cc2a63356-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.250950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.250922    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"]
Oct 30 01:06:17.295908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.290481    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:17.297364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.297265    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396f4826-7wc58"]
Oct 30 01:06:17.297364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.297282    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfc5s\" (UniqueName: \"kubernetes.io/projected/5775e7cb-9494-47be-b959-4a0cc2a63356-kube-api-access-sfc5s\") pod \"payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.312409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.312382    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84afeab-k76fh"]
Oct 30 01:06:17.486591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.486497    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:17.650784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.650751    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"]
Oct 30 01:06:17.660159 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:06:17.660126    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf627cdcb_aed4_496b_9137_2a8a60c48827.slice/crio-35a530545c4affc829161d31a7aecfafc5f37b71f745fafbeb9ec4a52fbecc95 WatchSource:0}: Error finding container 35a530545c4affc829161d31a7aecfafc5f37b71f745fafbeb9ec4a52fbecc95: Status 404 returned error can't find the container with id 35a530545c4affc829161d31a7aecfafc5f37b71f745fafbeb9ec4a52fbecc95
Oct 30 01:06:17.840930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:17.840893    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"]
Oct 30 01:06:18.280755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.280676    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"]
Oct 30 01:06:18.285713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.285690    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.299690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.299645    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:06:18.300058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.300025    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:06:18.300306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.300290    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:06:18.303961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.303745    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"]
Oct 30 01:06:18.307694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.307675    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:06:18.378429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.378381    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dd7\" (UniqueName: \"kubernetes.io/projected/2acb6698-620d-4434-b550-420b54026f3d-kube-api-access-v8dd7\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.378664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.378562    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acb6698-620d-4434-b550-420b54026f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.378753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.378696    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acb6698-620d-4434-b550-420b54026f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.405901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.405845    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" event={"ID":"f627cdcb-aed4-496b-9137-2a8a60c48827","Type":"ContainerStarted","Data":"219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5"}
Oct 30 01:06:18.406101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.405937    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:18.406101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.405961    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" event={"ID":"f627cdcb-aed4-496b-9137-2a8a60c48827","Type":"ContainerStarted","Data":"35a530545c4affc829161d31a7aecfafc5f37b71f745fafbeb9ec4a52fbecc95"}
Oct 30 01:06:18.408327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.408297    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" event={"ID":"5775e7cb-9494-47be-b959-4a0cc2a63356","Type":"ContainerStarted","Data":"61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703"}
Oct 30 01:06:18.408474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.408347    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" event={"ID":"5775e7cb-9494-47be-b959-4a0cc2a63356","Type":"ContainerStarted","Data":"b47fb7b4a46e9fbc2bae715f87da9ec6f82d1e32eb630c323838be1acc6abffc"}
Oct 30 01:06:18.408474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.408373    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:18.445153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.445102    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" podStartSLOduration=2.44508798 podStartE2EDuration="2.44508798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:18.442380217 +0000 UTC m=+1419.990070712" watchObservedRunningTime="2026-10-30 01:06:18.44508798 +0000 UTC m=+1419.992778476"
Oct 30 01:06:18.480996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.480927    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dd7\" (UniqueName: \"kubernetes.io/projected/2acb6698-620d-4434-b550-420b54026f3d-kube-api-access-v8dd7\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.481340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.481308    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acb6698-620d-4434-b550-420b54026f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.481557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.481431    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acb6698-620d-4434-b550-420b54026f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.482720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.482686    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acb6698-620d-4434-b550-420b54026f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.484525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.484475    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" podStartSLOduration=2.48444151 podStartE2EDuration="2.48444151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:18.478303807 +0000 UTC m=+1420.025994302" watchObservedRunningTime="2026-10-30 01:06:18.48444151 +0000 UTC m=+1420.032132033"
Oct 30 01:06:18.484676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.484603    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acb6698-620d-4434-b550-420b54026f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.501819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.501789    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dd7\" (UniqueName: \"kubernetes.io/projected/2acb6698-620d-4434-b550-420b54026f3d-kube-api-access-v8dd7\") pod \"maas-api-revoke-keys-e2e-mt-08065c8e-chhb2\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:18.583040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.582955    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c5470c-11cf-4b28-850f-d985b4b84cbb" path="/var/lib/kubelet/pods/a9c5470c-11cf-4b28-850f-d985b4b84cbb/volumes"
Oct 30 01:06:18.583652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.583622    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5039c56-329d-478e-926a-c9cf551e043f" path="/var/lib/kubelet/pods/b5039c56-329d-478e-926a-c9cf551e043f/volumes"
Oct 30 01:06:18.609558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:18.606773    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:19.029850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:19.029818    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"]
Oct 30 01:06:19.037515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:19.037443    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"]
Oct 30 01:06:19.037674 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:06:19.037648    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acb6698_620d_4434_b550_420b54026f3d.slice/crio-b76222d630ace569153be4e140b6af19f2615959c9035749cef97ec4748a4b06 WatchSource:0}: Error finding container b76222d630ace569153be4e140b6af19f2615959c9035749cef97ec4748a4b06: Status 404 returned error can't find the container with id b76222d630ace569153be4e140b6af19f2615959c9035749cef97ec4748a4b06
Oct 30 01:06:19.044201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:19.044178    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf40a09-c6btq"]
Oct 30 01:06:19.311834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:19.311796    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:06:19.420382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:19.420336    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2" event={"ID":"2acb6698-620d-4434-b550-420b54026f3d","Type":"ContainerStarted","Data":"b76222d630ace569153be4e140b6af19f2615959c9035749cef97ec4748a4b06"}
Oct 30 01:06:20.432147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:20.432101    5104 generic.go:356] "Generic (PLEG): container finished" podID="2acb6698-620d-4434-b550-420b54026f3d" containerID="5da5806efe89fd1f41748c986e754e8727ec6d203f064c93d4ca038a3b6da9f4" exitCode=0
Oct 30 01:06:20.432594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:20.432185    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2" event={"ID":"2acb6698-620d-4434-b550-420b54026f3d","Type":"ContainerDied","Data":"5da5806efe89fd1f41748c986e754e8727ec6d203f064c93d4ca038a3b6da9f4"}
Oct 30 01:06:20.583352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:20.583308    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4db46f-0cdf-4caf-a510-3db194a43a14" path="/var/lib/kubelet/pods/6c4db46f-0cdf-4caf-a510-3db194a43a14/volumes"
Oct 30 01:06:21.678813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.678782    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:21.760316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.760219    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2acb6698-620d-4434-b550-420b54026f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acb6698-620d-4434-b550-420b54026f3d-maas-api-openshift-ca\") pod \"2acb6698-620d-4434-b550-420b54026f3d\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") "
Oct 30 01:06:21.760550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.760380    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2acb6698-620d-4434-b550-420b54026f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acb6698-620d-4434-b550-420b54026f3d-maas-api-xks-ca\") pod \"2acb6698-620d-4434-b550-420b54026f3d\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") "
Oct 30 01:06:21.760681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.760602    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acb6698-620d-4434-b550-420b54026f3d-kube-api-access-v8dd7\" (UniqueName: \"kubernetes.io/projected/2acb6698-620d-4434-b550-420b54026f3d-kube-api-access-v8dd7\") pod \"2acb6698-620d-4434-b550-420b54026f3d\" (UID: \"2acb6698-620d-4434-b550-420b54026f3d\") "
Oct 30 01:06:21.760681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.760648    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acb6698-620d-4434-b550-420b54026f3d-maas-api-openshift-ca" pod "2acb6698-620d-4434-b550-420b54026f3d" (UID: "2acb6698-620d-4434-b550-420b54026f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:21.762117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.762089    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acb6698-620d-4434-b550-420b54026f3d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:21.763151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.763116    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acb6698-620d-4434-b550-420b54026f3d-kube-api-access-v8dd7" pod "2acb6698-620d-4434-b550-420b54026f3d" (UID: "2acb6698-620d-4434-b550-420b54026f3d"). InnerVolumeSpecName "kube-api-access-v8dd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:21.763259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.763200    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2acb6698-620d-4434-b550-420b54026f3d-maas-api-xks-ca" pod "2acb6698-620d-4434-b550-420b54026f3d" (UID: "2acb6698-620d-4434-b550-420b54026f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:21.864134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.864102    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dd7\" (UniqueName: \"kubernetes.io/projected/2acb6698-620d-4434-b550-420b54026f3d-kube-api-access-v8dd7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:21.864134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:21.864133    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acb6698-620d-4434-b550-420b54026f3d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:22.479295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:22.479264    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"
Oct 30 01:06:22.479513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:22.479292    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2" event={"ID":"2acb6698-620d-4434-b550-420b54026f3d","Type":"ContainerDied","Data":"b76222d630ace569153be4e140b6af19f2615959c9035749cef97ec4748a4b06"}
Oct 30 01:06:22.479513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:22.479353    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b76222d630ace569153be4e140b6af19f2615959c9035749cef97ec4748a4b06"
Oct 30 01:06:28.030383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:28.030342    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"]
Oct 30 01:06:28.041899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:28.041864    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf3e35d-tk52c"]
Oct 30 01:06:28.637835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:28.637794    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745dead4-1e2e-4a4f-9f67-d8a6326ffa59" path="/var/lib/kubelet/pods/745dead4-1e2e-4a4f-9f67-d8a6326ffa59/volumes"
Oct 30 01:06:28.663788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:28.663745    5104 generic.go:356] "Generic (PLEG): container finished" podID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerID="1faa4c2ff1e6f40e166761025c923be7b751062118b9cedb6afa2d9b5376ccca" exitCode=0
Oct 30 01:06:28.664038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:28.663807    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" event={"ID":"d708b71f-6a70-4a1d-a8e6-56cc9f56596d","Type":"ContainerDied","Data":"1faa4c2ff1e6f40e166761025c923be7b751062118b9cedb6afa2d9b5376ccca"}
Oct 30 01:06:29.028630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:29.028575    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"]
Oct 30 01:06:29.471473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:29.470480    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"]
Oct 30 01:06:29.517482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:29.490505    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:06:29.519395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:29.518246    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:06:30.072548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.060393    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:06:30.263481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.259220    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:06:30.359538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.342241    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:06:30.495543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.361358    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.161:50052\" within 1s: context deadline exceeded"
Oct 30 01:06:30.495543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.482706    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:06:30.495543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.473668    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:30.502750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.482938    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:30.527921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.475379    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:30.548969 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:06:30.397566    5104 logging.go:55] [core] [Channel #2243 SubChannel #2244]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.161:50052", ServerName: "10.132.0.161:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.161:50052: operation was canceled"
Oct 30 01:06:30.548969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.538018    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:06:30.548969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.548693    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:30.549300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.501925    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:30.608937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.608894    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-plugins-config-volume\") pod \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") "
Oct 30 01:06:30.609181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.609164    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-kube-api-access-k8ld9\" (UniqueName: \"kubernetes.io/projected/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-kube-api-access-k8ld9\") pod \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\" (UID: \"d708b71f-6a70-4a1d-a8e6-56cc9f56596d\") "
Oct 30 01:06:30.754299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.753742    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-plugins-config-volume" pod "d708b71f-6a70-4a1d-a8e6-56cc9f56596d" (UID: "d708b71f-6a70-4a1d-a8e6-56cc9f56596d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:30.929406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.918830    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-kube-api-access-k8ld9" pod "d708b71f-6a70-4a1d-a8e6-56cc9f56596d" (UID: "d708b71f-6a70-4a1d-a8e6-56cc9f56596d"). InnerVolumeSpecName "kube-api-access-k8ld9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:30.929406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.919339    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:30.952113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.952063    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:31.028407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:30.970576    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:31.059122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.022766    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ld9\" (UniqueName: \"kubernetes.io/projected/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-kube-api-access-k8ld9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:31.059415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.059397    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d708b71f-6a70-4a1d-a8e6-56cc9f56596d-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:31.074539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.070557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:06:31.074539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.071255    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:06:31.338645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.338541    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Oct 30 01:06:31.508879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.508839    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:06:31.532419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.513653    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:31.550481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.533677    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:06:31.602630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.602549    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:06:31.604737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.545773    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:06:31.896538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.553525    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:06:31.896538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.571323    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:31.896538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.550730    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:32.068550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:31.946494    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:32.068550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.028662    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:06:32.083524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.073792    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:32.573525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.541839    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:32.573525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.570279    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:32.573525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.571197    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:32.752491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.651679    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:06:32.799522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.766570    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:06:32.799522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.766639    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:32.960526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.937588    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:06:32.960526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:32.937658    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:33.077514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.071641    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:33.269291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.269179    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb" podUID="626e0a50-e9f7-4b0f-aa09-226cba9a0dfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.339388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.339332    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.340955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.340903    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.422782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.341158    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.431079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.341182    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.444711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.341215    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.455782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.382808    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:33.666725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.666605    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.756483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.701795    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.790487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.764947    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.788139    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:06:33.821162    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.169s"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.821318    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"]
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.853439    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2acb6698-620d-4434-b550-420b54026f3d" containerName="revoke-keys"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.853484    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acb6698-620d-4434-b550-420b54026f3d" containerName="revoke-keys"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.853607    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerName="payload-pre-processing"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.853616    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerName="payload-pre-processing"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.853840    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" containerName="payload-pre-processing"
Oct 30 01:06:33.869475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:33.853855    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acb6698-620d-4434-b550-420b54026f3d" containerName="revoke-keys"
Oct 30 01:06:34.069482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.065779    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks" podUID="73189491-ceaf-469c-951c-f22773bd9c23" containerName="payload-processing" containerID="cri-o://ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6" gracePeriod=25
Oct 30 01:06:34.133593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.092447    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.124492    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134064    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134093    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"]
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134142    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"]
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134161    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g" event={"ID":"d708b71f-6a70-4a1d-a8e6-56cc9f56596d","Type":"ContainerDied","Data":"c6f3624c6a56f7b4d8d7907eaf73d493cef5fd9a680ffbbfc4dca371edfe7005"}
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134229    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134271    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134311    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134351    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134386    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134424    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134478    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134513    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.134546    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.124585    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.161907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.143237    5104 scope.go:122] "RemoveContainer" containerID="1faa4c2ff1e6f40e166761025c923be7b751062118b9cedb6afa2d9b5376ccca"
Oct 30 01:06:34.163528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.162840    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:06:34.163528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.163040    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:06:34.163528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.163114    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:06:34.163528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.163191    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:06:34.278480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.211717    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.343539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.336357    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrttg\" (UniqueName: \"kubernetes.io/projected/5010c401-fa9d-4254-bccb-2b52cd111014-kube-api-access-vrttg\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.343539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.275765    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:06:34.343539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.338507    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abdd57be-tmfsl"]
Oct 30 01:06:34.354193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.354161    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.427637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.419248    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:06:34.456917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.456874    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:34.459216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.459180    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:06:34.462822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.462752    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.462958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.462894    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrttg\" (UniqueName: \"kubernetes.io/projected/5010c401-fa9d-4254-bccb-2b52cd111014-kube-api-access-vrttg\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.463094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.463046    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.465695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.464804    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.465947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.465933    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:06:34.470484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.470430    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.471144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.470896    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds" podUID="2189dd1e-6efe-408d-88ca-ae10099a0097" containerName="payload-pre-processing" containerID="cri-o://7fd5b4357d851893f3162c6619551bc35ba358f1971a134f85bf4b3655ff1d39" gracePeriod=25
Oct 30 01:06:34.530493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.511641    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrttg\" (UniqueName: \"kubernetes.io/projected/5010c401-fa9d-4254-bccb-2b52cd111014-kube-api-access-vrttg\") pod \"maas-api-revoke-sub-6c41a222-687cz\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.694596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.680049    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:34.816902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.804102    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925f65e7-de90-43a4-8903-e0e0e530d056" path="/var/lib/kubelet/pods/925f65e7-de90-43a4-8903-e0e0e530d056/volumes"
Oct 30 01:06:34.822186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:34.818040    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:06:35.062472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.062432    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:06:35.119259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.119217    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"]
Oct 30 01:06:35.126797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.126755    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-cvh5g"]
Oct 30 01:06:35.128653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.128619    5104 generic.go:356] "Generic (PLEG): container finished" podID="2189dd1e-6efe-408d-88ca-ae10099a0097" containerID="7fd5b4357d851893f3162c6619551bc35ba358f1971a134f85bf4b3655ff1d39" exitCode=0
Oct 30 01:06:35.128929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.128903    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds" event={"ID":"2189dd1e-6efe-408d-88ca-ae10099a0097","Type":"ContainerDied","Data":"7fd5b4357d851893f3162c6619551bc35ba358f1971a134f85bf4b3655ff1d39"}
Oct 30 01:06:35.136080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.136051    5104 generic.go:356] "Generic (PLEG): container finished" podID="73189491-ceaf-469c-951c-f22773bd9c23" containerID="ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6" exitCode=0
Oct 30 01:06:35.136219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.136102    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"
Oct 30 01:06:35.136297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.136252    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks" event={"ID":"73189491-ceaf-469c-951c-f22773bd9c23","Type":"ContainerDied","Data":"ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6"}
Oct 30 01:06:35.136297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.136293    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks" event={"ID":"73189491-ceaf-469c-951c-f22773bd9c23","Type":"ContainerDied","Data":"a9298db5c2cdab96a852d0723bc8f41752e796ebf654bc5212cabc420d748eeb"}
Oct 30 01:06:35.136398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.136317    5104 scope.go:122] "RemoveContainer" containerID="ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6"
Oct 30 01:06:35.189537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.169772    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73189491-ceaf-469c-951c-f22773bd9c23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73189491-ceaf-469c-951c-f22773bd9c23-plugins-config-volume\") pod \"73189491-ceaf-469c-951c-f22773bd9c23\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") "
Oct 30 01:06:35.189537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.170005    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73189491-ceaf-469c-951c-f22773bd9c23-kube-api-access-m89g6\" (UniqueName: \"kubernetes.io/projected/73189491-ceaf-469c-951c-f22773bd9c23-kube-api-access-m89g6\") pod \"73189491-ceaf-469c-951c-f22773bd9c23\" (UID: \"73189491-ceaf-469c-951c-f22773bd9c23\") "
Oct 30 01:06:35.189537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.179255    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73189491-ceaf-469c-951c-f22773bd9c23-plugins-config-volume" pod "73189491-ceaf-469c-951c-f22773bd9c23" (UID: "73189491-ceaf-469c-951c-f22773bd9c23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:35.189537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.188772    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:06:35.209541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.208867    5104 scope.go:122] "RemoveContainer" containerID="ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6"
Oct 30 01:06:35.209541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.209098    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73189491-ceaf-469c-951c-f22773bd9c23-kube-api-access-m89g6" pod "73189491-ceaf-469c-951c-f22773bd9c23" (UID: "73189491-ceaf-469c-951c-f22773bd9c23"). InnerVolumeSpecName "kube-api-access-m89g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:35.210551 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:06:35.210517    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6\": container with ID starting with ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6 not found: ID does not exist" containerID="ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6"
Oct 30 01:06:35.210656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.210570    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6"} err="failed to get container status \"ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6\": rpc error: code = NotFound desc = could not find container \"ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6\": container with ID starting with ee31f770c7e0bdda6b3682ba332cb71f25497b43cd86192722ce4e5324d62db6 not found: ID does not exist"
Oct 30 01:06:35.274956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.273034    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73189491-ceaf-469c-951c-f22773bd9c23-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:35.274956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.273066    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m89g6\" (UniqueName: \"kubernetes.io/projected/73189491-ceaf-469c-951c-f22773bd9c23-kube-api-access-m89g6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:35.376236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.376194    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2189dd1e-6efe-408d-88ca-ae10099a0097-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2189dd1e-6efe-408d-88ca-ae10099a0097-plugins-config-volume\") pod \"2189dd1e-6efe-408d-88ca-ae10099a0097\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") "
Oct 30 01:06:35.397768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.397730    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2189dd1e-6efe-408d-88ca-ae10099a0097-kube-api-access-4jtmq\" (UniqueName: \"kubernetes.io/projected/2189dd1e-6efe-408d-88ca-ae10099a0097-kube-api-access-4jtmq\") pod \"2189dd1e-6efe-408d-88ca-ae10099a0097\" (UID: \"2189dd1e-6efe-408d-88ca-ae10099a0097\") "
Oct 30 01:06:35.398644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.391724    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2189dd1e-6efe-408d-88ca-ae10099a0097-plugins-config-volume" pod "2189dd1e-6efe-408d-88ca-ae10099a0097" (UID: "2189dd1e-6efe-408d-88ca-ae10099a0097"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:35.399718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.399692    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2189dd1e-6efe-408d-88ca-ae10099a0097-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:35.408053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.408020    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2189dd1e-6efe-408d-88ca-ae10099a0097-kube-api-access-4jtmq" pod "2189dd1e-6efe-408d-88ca-ae10099a0097" (UID: "2189dd1e-6efe-408d-88ca-ae10099a0097"). InnerVolumeSpecName "kube-api-access-4jtmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:35.501820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.501784    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jtmq\" (UniqueName: \"kubernetes.io/projected/2189dd1e-6efe-408d-88ca-ae10099a0097-kube-api-access-4jtmq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:35.507125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.507099    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"]
Oct 30 01:06:35.513892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.513850    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"]
Oct 30 01:06:35.520073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:35.520048    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08065c8e-68f98d88c4-svnks"]
Oct 30 01:06:36.150498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.150443    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz" event={"ID":"5010c401-fa9d-4254-bccb-2b52cd111014","Type":"ContainerStarted","Data":"7b79417619cc5a8c462772fe7192825d359e8703053507cf52bdbcae3386742f"}
Oct 30 01:06:36.150951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.150531    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz" event={"ID":"5010c401-fa9d-4254-bccb-2b52cd111014","Type":"ContainerStarted","Data":"699bd592857ab2f0646081e83cdd485e0a921f195a3610602def7b1c12d48939"}
Oct 30 01:06:36.152344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.152319    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"
Oct 30 01:06:36.152518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.152345    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds" event={"ID":"2189dd1e-6efe-408d-88ca-ae10099a0097","Type":"ContainerDied","Data":"df17c05dd783c2ad21b5a9311fabec658b421fc0d4b4c9df10cc1460dd67d0d0"}
Oct 30 01:06:36.152518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.152411    5104 scope.go:122] "RemoveContainer" containerID="7fd5b4357d851893f3162c6619551bc35ba358f1971a134f85bf4b3655ff1d39"
Oct 30 01:06:36.206029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.205961    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz" podStartSLOduration=5.205943406 podStartE2EDuration="5.205943406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:36.194390996 +0000 UTC m=+1437.742081493" watchObservedRunningTime="2026-10-30 01:06:36.205943406 +0000 UTC m=+1437.753633902"
Oct 30 01:06:36.242589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.242546    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"]
Oct 30 01:06:36.248612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.248580    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08065c8e-67fbc9b5b7-m6bds"]
Oct 30 01:06:36.611766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.611733    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2189dd1e-6efe-408d-88ca-ae10099a0097" path="/var/lib/kubelet/pods/2189dd1e-6efe-408d-88ca-ae10099a0097/volumes"
Oct 30 01:06:36.612318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.612302    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73189491-ceaf-469c-951c-f22773bd9c23" path="/var/lib/kubelet/pods/73189491-ceaf-469c-951c-f22773bd9c23/volumes"
Oct 30 01:06:36.612758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:36.612744    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d708b71f-6a70-4a1d-a8e6-56cc9f56596d" path="/var/lib/kubelet/pods/d708b71f-6a70-4a1d-a8e6-56cc9f56596d/volumes"
Oct 30 01:06:37.165535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:37.165499    5104 generic.go:356] "Generic (PLEG): container finished" podID="5010c401-fa9d-4254-bccb-2b52cd111014" containerID="7b79417619cc5a8c462772fe7192825d359e8703053507cf52bdbcae3386742f" exitCode=0
Oct 30 01:06:37.165967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:37.165591    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz" event={"ID":"5010c401-fa9d-4254-bccb-2b52cd111014","Type":"ContainerDied","Data":"7b79417619cc5a8c462772fe7192825d359e8703053507cf52bdbcae3386742f"}
Oct 30 01:06:38.028535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.028503    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"]
Oct 30 01:06:38.033931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.033902    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-801fc8cf-rfckg"]
Oct 30 01:06:38.399903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.399874    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:38.477392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.477358    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-xks-ca\") pod \"5010c401-fa9d-4254-bccb-2b52cd111014\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") "
Oct 30 01:06:38.477629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.477484    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-openshift-ca\") pod \"5010c401-fa9d-4254-bccb-2b52cd111014\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") "
Oct 30 01:06:38.478073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.477795    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5010c401-fa9d-4254-bccb-2b52cd111014-kube-api-access-vrttg\" (UniqueName: \"kubernetes.io/projected/5010c401-fa9d-4254-bccb-2b52cd111014-kube-api-access-vrttg\") pod \"5010c401-fa9d-4254-bccb-2b52cd111014\" (UID: \"5010c401-fa9d-4254-bccb-2b52cd111014\") "
Oct 30 01:06:38.478572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.478544    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-openshift-ca" pod "5010c401-fa9d-4254-bccb-2b52cd111014" (UID: "5010c401-fa9d-4254-bccb-2b52cd111014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:38.481285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.481177    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5010c401-fa9d-4254-bccb-2b52cd111014-kube-api-access-vrttg" pod "5010c401-fa9d-4254-bccb-2b52cd111014" (UID: "5010c401-fa9d-4254-bccb-2b52cd111014"). InnerVolumeSpecName "kube-api-access-vrttg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:38.481285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.481245    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-xks-ca" pod "5010c401-fa9d-4254-bccb-2b52cd111014" (UID: "5010c401-fa9d-4254-bccb-2b52cd111014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:38.581489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.580575    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:38.581489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.580609    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrttg\" (UniqueName: \"kubernetes.io/projected/5010c401-fa9d-4254-bccb-2b52cd111014-kube-api-access-vrttg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:38.581489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.580624    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010c401-fa9d-4254-bccb-2b52cd111014-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:06:38.586606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:38.586573    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09307ada-bbab-4ea2-8c61-86e50a8d2c78" path="/var/lib/kubelet/pods/09307ada-bbab-4ea2-8c61-86e50a8d2c78/volumes"
Oct 30 01:06:39.033074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:39.032976    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"]
Oct 30 01:06:39.045003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:39.044960    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99376cb4-25l7k"]
Oct 30 01:06:39.413385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:39.413068    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz" event={"ID":"5010c401-fa9d-4254-bccb-2b52cd111014","Type":"ContainerDied","Data":"699bd592857ab2f0646081e83cdd485e0a921f195a3610602def7b1c12d48939"}
Oct 30 01:06:39.413385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:39.413130    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699bd592857ab2f0646081e83cdd485e0a921f195a3610602def7b1c12d48939"
Oct 30 01:06:39.413385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:39.413204    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"
Oct 30 01:06:39.659612 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:06:39.659566    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:06:40.585587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:40.585548    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf18d57-9466-4bc6-a645-08d241c4061f" path="/var/lib/kubelet/pods/cbf18d57-9466-4bc6-a645-08d241c4061f/volumes"
Oct 30 01:06:41.253130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.253095    5104 scope.go:122] "RemoveContainer" containerID="7208130a5f2bbe9e8f8164835cb928f0a4047b982c3764b0ca7de49712f7ad84"
Oct 30 01:06:41.276822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.276795    5104 scope.go:122] "RemoveContainer" containerID="55670830ea1d7afb1653e5f68b677d71200dc2b5d9251a9ffed312b9d12860a4"
Oct 30 01:06:41.300759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.300730    5104 scope.go:122] "RemoveContainer" containerID="5e7bed58f7b6aa65b71fa259e46b428ad35ebb65c15fe72568a50be45e7b3712"
Oct 30 01:06:41.411722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.392823    5104 scope.go:122] "RemoveContainer" containerID="bc3269130ebfaa514af493c84a3cf12048cb97090c27f67a89f3d70a744da1a9"
Oct 30 01:06:41.434679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.434652    5104 scope.go:122] "RemoveContainer" containerID="ef2de4f7048be0ba53d2e4f508c4552816d614fd27c96f6d837e9b1ea2f7303d"
Oct 30 01:06:41.453205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.453183    5104 scope.go:122] "RemoveContainer" containerID="805db9de441ff5c7eac22b8e05898270a8260925f60e40e8b8bc986fffda0df4"
Oct 30 01:06:41.465138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.465115    5104 scope.go:122] "RemoveContainer" containerID="b73222d179b5c4346b4d211db98fea5849ead8bb0e3b84b2ce55bac62f5091dc"
Oct 30 01:06:41.479553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.479528    5104 scope.go:122] "RemoveContainer" containerID="9084ed1a0b51093158b477c3214ff5f8ad7dbed4843137cb78845d28386f6012"
Oct 30 01:06:41.493970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.493946    5104 scope.go:122] "RemoveContainer" containerID="65d1d0987626f31eb6ab6b6917935b63b9423c6cf5b0768343730fcb7ce1f336"
Oct 30 01:06:41.509295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:41.509269    5104 scope.go:122] "RemoveContainer" containerID="9d22e45c1e1abddc2c73cde277b13a5c94d079e5c124b9144101066a5be6a13b"
Oct 30 01:06:42.023650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.023609    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:06:42.024043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.023680    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:42.031643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.031595    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"]
Oct 30 01:06:42.057553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.040534    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d512610-kfzn7"]
Oct 30 01:06:42.460236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.460202    5104 generic.go:356] "Generic (PLEG): container finished" podID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerID="8b913aab1f84d28d56ead6e21e01913b007640830de5fb1aa93f591200d44ace" exitCode=0
Oct 30 01:06:42.460420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.460276    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" event={"ID":"a76516bd-ba32-46c7-8a0f-878527bc1633","Type":"ContainerDied","Data":"8b913aab1f84d28d56ead6e21e01913b007640830de5fb1aa93f591200d44ace"}
Oct 30 01:06:42.460866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.460847    5104 scope.go:122] "RemoveContainer" containerID="8b913aab1f84d28d56ead6e21e01913b007640830de5fb1aa93f591200d44ace"
Oct 30 01:06:42.463133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.463094    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:42.463260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.463216    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:42.463326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.463270    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:42.634968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:42.634918    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d70886c-cc98-4c03-84f1-4cfdd6e447db" path="/var/lib/kubelet/pods/2d70886c-cc98-4c03-84f1-4cfdd6e447db/volumes"
Oct 30 01:06:43.068482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:43.067509    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"]
Oct 30 01:06:43.097479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:43.090036    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba82ac8-jxmp7"]
Oct 30 01:06:43.473807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:43.473771    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" event={"ID":"a76516bd-ba32-46c7-8a0f-878527bc1633","Type":"ContainerStarted","Data":"5d16ccf4848fc7e79b0f97af660da0fe5adce933b14a9ec6d118a6fc92272d16"}
Oct 30 01:06:44.130478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.129765    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:06:44.130478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.129884    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:06:44.130478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.129939    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.450797    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"]
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452161    5104 cpu_manager.go:414] "containerMap: removing container" podUID="73189491-ceaf-469c-951c-f22773bd9c23" containerName="payload-processing"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452182    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73189491-ceaf-469c-951c-f22773bd9c23" containerName="payload-processing"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452212    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5010c401-fa9d-4254-bccb-2b52cd111014" containerName="revoke-keys"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452221    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5010c401-fa9d-4254-bccb-2b52cd111014" containerName="revoke-keys"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452260    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2189dd1e-6efe-408d-88ca-ae10099a0097" containerName="payload-pre-processing"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452270    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2189dd1e-6efe-408d-88ca-ae10099a0097" containerName="payload-pre-processing"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452517    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="73189491-ceaf-469c-951c-f22773bd9c23" containerName="payload-processing"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452534    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5010c401-fa9d-4254-bccb-2b52cd111014" containerName="revoke-keys"
Oct 30 01:06:44.455133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.452551    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2189dd1e-6efe-408d-88ca-ae10099a0097" containerName="payload-pre-processing"
Oct 30 01:06:44.458258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.458237    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.486547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.472439    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"]
Oct 30 01:06:44.493714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.493672    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-07410c-openshift-default-dockercfg-f9tgq\""
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.666985    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-envoy\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667133    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm48k\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-kube-api-access-fm48k\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667201    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-socket\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667259    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-istio-token\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667380    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc10b078-6065-4c3b-9655-306808b43f55-istiod-ca-cert\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667496    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-certs\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667720    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-credential-socket\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667753    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc10b078-6065-4c3b-9655-306808b43f55-istio-podinfo\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.691029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.667867    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-data\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.771270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.771181    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm48k\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-kube-api-access-fm48k\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.779842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.779811    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-socket\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.792506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.787127    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-socket\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.797547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.797498    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-istio-token\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.797920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.797895    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc10b078-6065-4c3b-9655-306808b43f55-istiod-ca-cert\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.806489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.798109    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-certs\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.814487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.806930    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-credential-socket\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.814791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.814759    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc10b078-6065-4c3b-9655-306808b43f55-istio-podinfo\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.819687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.819636    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-data\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.823479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.819968    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-envoy\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.851587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.802676    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc10b078-6065-4c3b-9655-306808b43f55-istiod-ca-cert\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.860637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.802022    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-certs\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.861276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.861246    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-credential-socket\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.894338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.882776    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-data\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:44.946270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:44.938074    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc10b078-6065-4c3b-9655-306808b43f55-istio-podinfo\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:45.042793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.042676    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-envoy\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:45.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.173683    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:06:45.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.173759    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:06:45.266827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.266724    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm48k\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-kube-api-access-fm48k\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:45.292533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.217442    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c081df0-12e0-4e96-864b-60f9aa88a732" path="/var/lib/kubelet/pods/7c081df0-12e0-4e96-864b-60f9aa88a732/volumes"
Oct 30 01:06:45.389544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.371072    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-istio-token\") pod \"e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") " pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:45.424183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.424146    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:45.555966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.555212    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:45.649079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.648982    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:45.656475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:45.651338    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:46.024548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:46.021565    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"]
Oct 30 01:06:46.031139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:46.031095    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:46.031274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:46.031219    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:46.031334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:46.031280    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:06:46.118816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:46.118763    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" event={"ID":"fc10b078-6065-4c3b-9655-306808b43f55","Type":"ContainerStarted","Data":"96e538ce8ab35e1269f39f922f4fd78f4adc126a44f6ec5075cefd7a509f0de8"}
Oct 30 01:06:46.129493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:46.129449    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:06:46.129700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:46.129680    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:06:47.129544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:47.129505    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:06:47.129951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:47.129580    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:06:47.132583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:47.132554    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" event={"ID":"fc10b078-6065-4c3b-9655-306808b43f55","Type":"ContainerStarted","Data":"93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe"}
Oct 30 01:06:47.205181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:47.205102    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podStartSLOduration=3.2050762600000002 podStartE2EDuration="3.20507626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:47.187798273 +0000 UTC m=+1448.735488770" watchObservedRunningTime="2026-10-30 01:06:47.20507626 +0000 UTC m=+1448.752766757"
Oct 30 01:06:47.431606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:47.428896    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:47.457932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:47.457887    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:48.142241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:48.142207    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:06:48.142843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:48.142810    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:06:48.228947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:48.227084    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:48.230914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:48.230870    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:06:49.129686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:49.129652    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:06:49.129871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:49.129707    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:06:50.130149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:50.130110    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:06:50.130608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:50.130180    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:06:50.918169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:50.918132    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:06:51.593401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:51.593364    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:06:51.595331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:51.595186    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:06:51.598188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:51.598130    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:06:52.051602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.051564    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"]
Oct 30 01:06:52.056914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.056886    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16cd7001-2gskp"]
Oct 30 01:06:52.113103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.113069    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"]
Oct 30 01:06:52.119934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.119907    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.135523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.135491    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"]
Oct 30 01:06:52.136743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.136719    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-07410c\""
Oct 30 01:06:52.136912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.136896    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-07410c-dockercfg-q789m\""
Oct 30 01:06:52.295106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.272875    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4frg\" (UniqueName: \"kubernetes.io/projected/96558580-ea47-4d9a-851c-8b2ef823a82b-kube-api-access-s4frg\") pod \"payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.295449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.295418    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96558580-ea47-4d9a-851c-8b2ef823a82b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.558004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.555919    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4frg\" (UniqueName: \"kubernetes.io/projected/96558580-ea47-4d9a-851c-8b2ef823a82b-kube-api-access-s4frg\") pod \"payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.558004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.555969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96558580-ea47-4d9a-851c-8b2ef823a82b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.558004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.557288    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96558580-ea47-4d9a-851c-8b2ef823a82b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.827531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.769195    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4frg\" (UniqueName: \"kubernetes.io/projected/96558580-ea47-4d9a-851c-8b2ef823a82b-kube-api-access-s4frg\") pod \"payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.827531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.784013    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:52.972299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.972037    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e185f0b-b201-4239-b7c6-a7a00374f957" path="/var/lib/kubelet/pods/2e185f0b-b201-4239-b7c6-a7a00374f957/volumes"
Oct 30 01:06:52.973522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.972718    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"]
Oct 30 01:06:52.981924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.981655    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"]
Oct 30 01:06:52.982079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:52.982026    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.106087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.081407    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98pq\" (UniqueName: \"kubernetes.io/projected/80c0c049-e530-4089-b133-00d5b8a695d2-kube-api-access-c98pq\") pod \"payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.106087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.082905    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c0c049-e530-4089-b133-00d5b8a695d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.283874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.211094    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c0c049-e530-4089-b133-00d5b8a695d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.283874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.211491    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98pq\" (UniqueName: \"kubernetes.io/projected/80c0c049-e530-4089-b133-00d5b8a695d2-kube-api-access-c98pq\") pod \"payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.283874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.212778    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c0c049-e530-4089-b133-00d5b8a695d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.485486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.477161    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98pq\" (UniqueName: \"kubernetes.io/projected/80c0c049-e530-4089-b133-00d5b8a695d2-kube-api-access-c98pq\") pod \"payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.485486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.483985    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:53.738280 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:06:53.738219    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96558580_ea47_4d9a_851c_8b2ef823a82b.slice/crio-ed6a492297ddc7d6af0c213dbcc0925aa7ec5d301e694ea77caa40dbbfbb38b9 WatchSource:0}: Error finding container ed6a492297ddc7d6af0c213dbcc0925aa7ec5d301e694ea77caa40dbbfbb38b9: Status 404 returned error can't find the container with id ed6a492297ddc7d6af0c213dbcc0925aa7ec5d301e694ea77caa40dbbfbb38b9
Oct 30 01:06:53.739708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.739680    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"]
Oct 30 01:06:53.813583 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:06:53.813538    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c0c049_e530_4089_b133_00d5b8a695d2.slice/crio-a8c1e240b7635f54378fa5a65e23e789f8c11bd2ac072701d3756378b2dcd9fa WatchSource:0}: Error finding container a8c1e240b7635f54378fa5a65e23e789f8c11bd2ac072701d3756378b2dcd9fa: Status 404 returned error can't find the container with id a8c1e240b7635f54378fa5a65e23e789f8c11bd2ac072701d3756378b2dcd9fa
Oct 30 01:06:53.815814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.815790    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"]
Oct 30 01:06:53.872177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.872140    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm" event={"ID":"80c0c049-e530-4089-b133-00d5b8a695d2","Type":"ContainerStarted","Data":"a8c1e240b7635f54378fa5a65e23e789f8c11bd2ac072701d3756378b2dcd9fa"}
Oct 30 01:06:53.874198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.874169    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp" event={"ID":"96558580-ea47-4d9a-851c-8b2ef823a82b","Type":"ContainerStarted","Data":"7369f40e80d919fc09aa52a7bd70a773801cb397d2267ad3b640602aee71972f"}
Oct 30 01:06:53.874378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.874221    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp" event={"ID":"96558580-ea47-4d9a-851c-8b2ef823a82b","Type":"ContainerStarted","Data":"ed6a492297ddc7d6af0c213dbcc0925aa7ec5d301e694ea77caa40dbbfbb38b9"}
Oct 30 01:06:53.874378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.874269    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:06:53.913630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:53.913587    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp" podStartSLOduration=1.913573958 podStartE2EDuration="1.913573958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:53.910903275 +0000 UTC m=+1455.458593771" watchObservedRunningTime="2026-10-30 01:06:53.913573958 +0000 UTC m=+1455.461264491"
Oct 30 01:06:55.039844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:55.039721    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm" event={"ID":"80c0c049-e530-4089-b133-00d5b8a695d2","Type":"ContainerStarted","Data":"4b0d997b26827b61d5e2b13ded9f17c947e44f94ef5200407451a1fd1224d573"}
Oct 30 01:06:55.039844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:55.039804    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:06:55.098660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:06:55.098591    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm" podStartSLOduration=3.09856927 podStartE2EDuration="3.09856927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:55.087097992 +0000 UTC m=+1456.634788487" watchObservedRunningTime="2026-10-30 01:06:55.09856927 +0000 UTC m=+1456.646259769"
Oct 30 01:07:00.933703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:00.933661    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"]
Oct 30 01:07:00.946821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:00.946167    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08065c8e-chhb2"]
Oct 30 01:07:02.029444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:02.029405    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"]
Oct 30 01:07:02.040917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:02.040884    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfac88a7-xj7mh"]
Oct 30 01:07:02.580534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:02.580366    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1058d6ce-bd7b-4fdc-8e20-1272a92d3c38" path="/var/lib/kubelet/pods/1058d6ce-bd7b-4fdc-8e20-1272a92d3c38/volumes"
Oct 30 01:07:02.581081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:02.581056    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acb6698-620d-4434-b550-420b54026f3d" path="/var/lib/kubelet/pods/2acb6698-620d-4434-b550-420b54026f3d/volumes"
Oct 30 01:07:04.560406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.560372    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"]
Oct 30 01:07:04.566749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.566725    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.587336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.587303    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"]
Oct 30 01:07:04.589699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.589672    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:04.589858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.589748    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:07:04.589858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.589822    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:07:04.591621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.591602    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:07:04.603366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.602387    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.611499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.608449    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqhp\" (UniqueName: \"kubernetes.io/projected/691f9f06-7ff8-4031-ae6f-cb574e69c21b-kube-api-access-fvqhp\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.611499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.608609    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.710631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.710577    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqhp\" (UniqueName: \"kubernetes.io/projected/691f9f06-7ff8-4031-ae6f-cb574e69c21b-kube-api-access-fvqhp\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.711376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.711330    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.712724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.712698    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.713538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.712515    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.760058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.758865    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqhp\" (UniqueName: \"kubernetes.io/projected/691f9f06-7ff8-4031-ae6f-cb574e69c21b-kube-api-access-fvqhp\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.760058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.759302    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d855-79snv\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:04.890365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:04.890260    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:05.042306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:05.042277    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:07:05.225255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:05.225224    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"]
Oct 30 01:07:05.225670 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:05.225590    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691f9f06_7ff8_4031_ae6f_cb574e69c21b.slice/crio-eb06f718be9d1ea9ef70e1e27ce3a36a0d3d937370578956f77781ae811c390c WatchSource:0}: Error finding container eb06f718be9d1ea9ef70e1e27ce3a36a0d3d937370578956f77781ae811c390c: Status 404 returned error can't find the container with id eb06f718be9d1ea9ef70e1e27ce3a36a0d3d937370578956f77781ae811c390c
Oct 30 01:07:05.507905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:05.507812    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv" event={"ID":"691f9f06-7ff8-4031-ae6f-cb574e69c21b","Type":"ContainerStarted","Data":"eb06f718be9d1ea9ef70e1e27ce3a36a0d3d937370578956f77781ae811c390c"}
Oct 30 01:07:06.662758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:06.662724    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv" event={"ID":"691f9f06-7ff8-4031-ae6f-cb574e69c21b","Type":"ContainerStarted","Data":"605218bf9d18199485b8026066678980c8420d96fb676e77bcd8b9904c4ec134"}
Oct 30 01:07:07.063394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.063364    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:07:07.438635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.434030    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"]
Oct 30 01:07:07.438635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.434303    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy" containerID="cri-o://2650eb0e427f4305faf7dfa9ecd3b2d84f2c1509876635546273c560f4d15e86" gracePeriod=30
Oct 30 01:07:07.468752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.453329    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"]
Oct 30 01:07:07.468752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.453653    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy" containerID="cri-o://c82557edd652cbc19c56abd0c2fa5027e3d76fc62515d9b4e0c1ad22649d851f" gracePeriod=30
Oct 30 01:07:07.496483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.493089    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"]
Oct 30 01:07:07.496483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.493374    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy" containerID="cri-o://ce05a6d0d3bd11cf03f795e157a1df6ea1b9562270fc7a2923d7827aae7286e6" gracePeriod=30
Oct 30 01:07:07.504077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.502097    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"]
Oct 30 01:07:07.504077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.502339    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy" containerID="cri-o://360144cf31c29e09b299d9ab27b8628fc9edbab320fdeb488478cdee530c90dd" gracePeriod=30
Oct 30 01:07:07.989800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.989763    5104 generic.go:356] "Generic (PLEG): container finished" podID="691f9f06-7ff8-4031-ae6f-cb574e69c21b" containerID="605218bf9d18199485b8026066678980c8420d96fb676e77bcd8b9904c4ec134" exitCode=0
Oct 30 01:07:07.990999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:07.990660    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv" event={"ID":"691f9f06-7ff8-4031-ae6f-cb574e69c21b","Type":"ContainerDied","Data":"605218bf9d18199485b8026066678980c8420d96fb676e77bcd8b9904c4ec134"}
Oct 30 01:07:09.247490 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:07:09.245566    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:07:09.718309 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:07:09.718094    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.105s"
Oct 30 01:07:09.718594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.718564    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv" event={"ID":"691f9f06-7ff8-4031-ae6f-cb574e69c21b","Type":"ContainerDied","Data":"eb06f718be9d1ea9ef70e1e27ce3a36a0d3d937370578956f77781ae811c390c"}
Oct 30 01:07:09.718765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.718746    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb06f718be9d1ea9ef70e1e27ce3a36a0d3d937370578956f77781ae811c390c"
Oct 30 01:07:09.718966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.718953    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:09.795482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.768756    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691f9f06-7ff8-4031-ae6f-cb574e69c21b-kube-api-access-fvqhp\" (UniqueName: \"kubernetes.io/projected/691f9f06-7ff8-4031-ae6f-cb574e69c21b-kube-api-access-fvqhp\") pod \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") "
Oct 30 01:07:09.808787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.805582    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-xks-ca\") pod \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") "
Oct 30 01:07:09.808787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.805814    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-openshift-ca\") pod \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\" (UID: \"691f9f06-7ff8-4031-ae6f-cb574e69c21b\") "
Oct 30 01:07:09.852163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.832081    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691f9f06-7ff8-4031-ae6f-cb574e69c21b-kube-api-access-fvqhp" pod "691f9f06-7ff8-4031-ae6f-cb574e69c21b" (UID: "691f9f06-7ff8-4031-ae6f-cb574e69c21b"). InnerVolumeSpecName "kube-api-access-fvqhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:09.858738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.858698    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-openshift-ca" pod "691f9f06-7ff8-4031-ae6f-cb574e69c21b" (UID: "691f9f06-7ff8-4031-ae6f-cb574e69c21b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:09.892680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.882116    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"]
Oct 30 01:07:09.922193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.922162    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqhp\" (UniqueName: \"kubernetes.io/projected/691f9f06-7ff8-4031-ae6f-cb574e69c21b-kube-api-access-fvqhp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:09.922446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.922425    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:09.968563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:09.965660    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-xks-ca" pod "691f9f06-7ff8-4031-ae6f-cb574e69c21b" (UID: "691f9f06-7ff8-4031-ae6f-cb574e69c21b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:10.025816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:10.025330    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/691f9f06-7ff8-4031-ae6f-cb574e69c21b-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:10.034313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:10.034286    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-9rnvn"]
Oct 30 01:07:10.064606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:10.064511    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"]
Oct 30 01:07:10.069634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:10.068272    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adacf3e4-tnbb5"]
Oct 30 01:07:10.638290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:10.638244    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b033b4a-f168-4a05-ab18-4c605db015e0" path="/var/lib/kubelet/pods/6b033b4a-f168-4a05-ab18-4c605db015e0/volumes"
Oct 30 01:07:10.639103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:10.639075    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ae3b79-af04-48db-a856-34ab58bd91bd" path="/var/lib/kubelet/pods/c5ae3b79-af04-48db-a856-34ab58bd91bd/volumes"
Oct 30 01:07:10.687589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:10.687559    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"
Oct 30 01:07:12.933940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.933904    5104 generic.go:356] "Generic (PLEG): container finished" podID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerID="ce05a6d0d3bd11cf03f795e157a1df6ea1b9562270fc7a2923d7827aae7286e6" exitCode=0
Oct 30 01:07:12.934395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.934124    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" event={"ID":"d00e8adc-8578-4335-86db-dd05f44ff6c4","Type":"ContainerDied","Data":"ce05a6d0d3bd11cf03f795e157a1df6ea1b9562270fc7a2923d7827aae7286e6"}
Oct 30 01:07:12.939690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.936096    5104 generic.go:356] "Generic (PLEG): container finished" podID="14d96c09-fa3c-419a-8036-7189095b73e4" containerID="2650eb0e427f4305faf7dfa9ecd3b2d84f2c1509876635546273c560f4d15e86" exitCode=0
Oct 30 01:07:12.939690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.936319    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" event={"ID":"14d96c09-fa3c-419a-8036-7189095b73e4","Type":"ContainerDied","Data":"2650eb0e427f4305faf7dfa9ecd3b2d84f2c1509876635546273c560f4d15e86"}
Oct 30 01:07:12.939690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.938932    5104 generic.go:356] "Generic (PLEG): container finished" podID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerID="360144cf31c29e09b299d9ab27b8628fc9edbab320fdeb488478cdee530c90dd" exitCode=0
Oct 30 01:07:12.939690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.939115    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" event={"ID":"84b90433-33b5-49ee-9cd5-2faf0655badd","Type":"ContainerDied","Data":"360144cf31c29e09b299d9ab27b8628fc9edbab320fdeb488478cdee530c90dd"}
Oct 30 01:07:12.939690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.939167    5104 scope.go:122] "RemoveContainer" containerID="4c3ee4fad64fa2c9c765525dc9a6cd2b2355a9432c98a0af326ec7d4a9c115e8"
Oct 30 01:07:12.945734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.945188    5104 generic.go:356] "Generic (PLEG): container finished" podID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerID="c82557edd652cbc19c56abd0c2fa5027e3d76fc62515d9b4e0c1ad22649d851f" exitCode=0
Oct 30 01:07:12.945734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.945202    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" event={"ID":"7f0b5121-f9ca-4203-9a44-ad3980d3f439","Type":"ContainerDied","Data":"c82557edd652cbc19c56abd0c2fa5027e3d76fc62515d9b4e0c1ad22649d851f"}
Oct 30 01:07:12.945734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:12.945704    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002370    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-certs\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002430    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-socket\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002637    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-credential-socket\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002702    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-envoy\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002770    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-data\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002842    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-kube-api-access-h856j\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-kube-api-access-h856j\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002910    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d96c09-fa3c-419a-8036-7189095b73e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d96c09-fa3c-419a-8036-7189095b73e4-istiod-ca-cert\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002926    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-certs" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.002958    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-istio-token\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-istio-token\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.003017    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14d96c09-fa3c-419a-8036-7189095b73e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d96c09-fa3c-419a-8036-7189095b73e4-istio-podinfo\") pod \"14d96c09-fa3c-419a-8036-7189095b73e4\" (UID: \"14d96c09-fa3c-419a-8036-7189095b73e4\") "
Oct 30 01:07:13.003757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.003513    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-data" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.004730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.003879    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-socket" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.004730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.004062    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-credential-socket" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.010004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.009830    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d96c09-fa3c-419a-8036-7189095b73e4-istiod-ca-cert" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:13.011202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.011170    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.011202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.011204    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.011407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.011221    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d96c09-fa3c-419a-8036-7189095b73e4-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.011407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.011237    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.011407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.011252    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.020522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.017681    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-kube-api-access-h856j" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "kube-api-access-h856j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.028918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.028737    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-envoy" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.028918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.028886    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-istio-token" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.029166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.029139    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14d96c09-fa3c-419a-8036-7189095b73e4-istio-podinfo" pod "14d96c09-fa3c-419a-8036-7189095b73e4" (UID: "14d96c09-fa3c-419a-8036-7189095b73e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:13.113527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.113497    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d96c09-fa3c-419a-8036-7189095b73e4-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.113527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.113525    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h856j\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-kube-api-access-h856j\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.113762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.113539    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d96c09-fa3c-419a-8036-7189095b73e4-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.113762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.113553    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d96c09-fa3c-419a-8036-7189095b73e4-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.124214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.124122    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.229863    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-podinfo\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.229927    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-socket\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.230025    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-certs\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.230117    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istiod-ca-cert\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.230214    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-envoy\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.230338    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-credential-socket\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.230421    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-data\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.230594    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-kube-api-access-rrzgx\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-kube-api-access-rrzgx\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.235503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.230668    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-token\") pod \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\" (UID: \"7f0b5121-f9ca-4203-9a44-ad3980d3f439\") "
Oct 30 01:07:13.264239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.262140    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-data" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.264239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.262312    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-credential-socket" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.275361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.274573    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-certs" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.275361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.275183    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-socket" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.275361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.275322    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istiod-ca-cert" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:13.338483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.317884    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 01:07:13.338483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.331729    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-token" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.357356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.348534    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.357356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.348571    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.357356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.348585    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.357356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.348601    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.357356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.348615    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.357356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.348628    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.412540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.370620    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-podinfo" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:13.412540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.378782    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-kube-api-access-rrzgx" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "kube-api-access-rrzgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.426101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.413819    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-envoy" pod "7f0b5121-f9ca-4203-9a44-ad3980d3f439" (UID: "7f0b5121-f9ca-4203-9a44-ad3980d3f439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450450    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-socket\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450537    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-kube-api-access-wt2xc\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-kube-api-access-wt2xc\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450608    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-envoy\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450685    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-data\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450720    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-credential-socket\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450764    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00e8adc-8578-4335-86db-dd05f44ff6c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e8adc-8578-4335-86db-dd05f44ff6c4-istiod-ca-cert\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450805    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-certs\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.450999    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-token\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-token\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.451130    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-podinfo\") pod \"d00e8adc-8578-4335-86db-dd05f44ff6c4\" (UID: \"d00e8adc-8578-4335-86db-dd05f44ff6c4\") "
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.451955    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.451979    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzgx\" (UniqueName: \"kubernetes.io/projected/7f0b5121-f9ca-4203-9a44-ad3980d3f439-kube-api-access-rrzgx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.451996    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b5121-f9ca-4203-9a44-ad3980d3f439-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.505187    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-data" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.509203    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-socket" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.531380    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00e8adc-8578-4335-86db-dd05f44ff6c4-istiod-ca-cert" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.531763    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-credential-socket" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.532162    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-certs" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.544002    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-envoy" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.554418    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.554449    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.554479    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.554493    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.554508    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e8adc-8578-4335-86db-dd05f44ff6c4-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.590526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.554521    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e8adc-8578-4335-86db-dd05f44ff6c4-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.592402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.590721    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-token" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.672015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.671965    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-podinfo" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:13.672495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.672426    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-kube-api-access-wt2xc" pod "d00e8adc-8578-4335-86db-dd05f44ff6c4" (UID: "d00e8adc-8578-4335-86db-dd05f44ff6c4"). InnerVolumeSpecName "kube-api-access-wt2xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.683888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.683587    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2xc\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-kube-api-access-wt2xc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.683888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.683627    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.683888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.683644    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e8adc-8578-4335-86db-dd05f44ff6c4-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.739001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.738614    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:07:13.887801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.887716    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-credential-socket\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.887801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.887784    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-podinfo\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.887851    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-data\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.887901    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-socket\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888001    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-kube-api-access-ldw5q\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-kube-api-access-ldw5q\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888043    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-credential-socket" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.888228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888053    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-certs\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888220    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-data" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.888333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888271    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-certs" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.888333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888310    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b90433-33b5-49ee-9cd5-2faf0655badd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b90433-33b5-49ee-9cd5-2faf0655badd-istiod-ca-cert\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888356    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-token\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-token\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888422    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-envoy\") pod \"84b90433-33b5-49ee-9cd5-2faf0655badd\" (UID: \"84b90433-33b5-49ee-9cd5-2faf0655badd\") "
Oct 30 01:07:13.888656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.888605    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-socket" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.889036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.889004    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b90433-33b5-49ee-9cd5-2faf0655badd-istiod-ca-cert" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:13.889345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.889320    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.889527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.889505    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.889527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.889528    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.889730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.889542    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b90433-33b5-49ee-9cd5-2faf0655badd-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.889730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.889555    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.890723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.890702    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-kube-api-access-ldw5q" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "kube-api-access-ldw5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.890843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.890771    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-podinfo" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:13.890843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.890836    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-token" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:13.891224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.891200    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-envoy" pod "84b90433-33b5-49ee-9cd5-2faf0655badd" (UID: "84b90433-33b5-49ee-9cd5-2faf0655badd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:13.958684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.958654    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"
Oct 30 01:07:13.959191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.958653    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5" event={"ID":"7f0b5121-f9ca-4203-9a44-ad3980d3f439","Type":"ContainerDied","Data":"1eaa9cb77473566278b837a0ef08307e03e5bc33599a73f30caee37f905d3199"}
Oct 30 01:07:13.959191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.958858    5104 scope.go:122] "RemoveContainer" containerID="c82557edd652cbc19c56abd0c2fa5027e3d76fc62515d9b4e0c1ad22649d851f"
Oct 30 01:07:13.964277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.960488    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"
Oct 30 01:07:13.964277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.960512    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p" event={"ID":"d00e8adc-8578-4335-86db-dd05f44ff6c4","Type":"ContainerDied","Data":"fd7d378f06132bfe0b50f93f0e3ccaf06f94849b1be5632b9413f435abe01a78"}
Oct 30 01:07:13.964277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.962294    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"
Oct 30 01:07:13.964277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.962311    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl" event={"ID":"14d96c09-fa3c-419a-8036-7189095b73e4","Type":"ContainerDied","Data":"436348ecca89789af97b7a9a41975bc619e286e36854f194808091229f51e4a8"}
Oct 30 01:07:13.964586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.964314    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"
Oct 30 01:07:13.964586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.964340    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9" event={"ID":"84b90433-33b5-49ee-9cd5-2faf0655badd","Type":"ContainerDied","Data":"8c2e90c012b7b315829540e269e3c81239da9b37019844fe22d2f136270920cf"}
Oct 30 01:07:13.980904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.980884    5104 scope.go:122] "RemoveContainer" containerID="ce05a6d0d3bd11cf03f795e157a1df6ea1b9562270fc7a2923d7827aae7286e6"
Oct 30 01:07:13.991225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.991198    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldw5q\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-kube-api-access-ldw5q\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.991337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.991231    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.991337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.991247    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.991337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.991260    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b90433-33b5-49ee-9cd5-2faf0655badd-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:13.996898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:13.996879    5104 scope.go:122] "RemoveContainer" containerID="2650eb0e427f4305faf7dfa9ecd3b2d84f2c1509876635546273c560f4d15e86"
Oct 30 01:07:14.010655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.010626    5104 scope.go:122] "RemoveContainer" containerID="360144cf31c29e09b299d9ab27b8628fc9edbab320fdeb488478cdee530c90dd"
Oct 30 01:07:14.023401 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:07:14.023358    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0b5121_f9ca_4203_9a44_ad3980d3f439.slice/crio-1eaa9cb77473566278b837a0ef08307e03e5bc33599a73f30caee37f905d3199\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00e8adc_8578_4335_86db_dd05f44ff6c4.slice/crio-fd7d378f06132bfe0b50f93f0e3ccaf06f94849b1be5632b9413f435abe01a78\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b90433_33b5_49ee_9cd5_2faf0655badd.slice/crio-8c2e90c012b7b315829540e269e3c81239da9b37019844fe22d2f136270920cf\": RecentStats: unable to find data in memory cache]"
Oct 30 01:07:14.059929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.059898    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"]
Oct 30 01:07:14.065967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.065930    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-q9dm5"]
Oct 30 01:07:14.097554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.097523    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"]
Oct 30 01:07:14.106859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.106817    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-j7lcl"]
Oct 30 01:07:14.138555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.138450    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"]
Oct 30 01:07:14.143705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.143674    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08065c8e-openshift-default-6f958bfddf-jqkv9"]
Oct 30 01:07:14.176431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.176394    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"]
Oct 30 01:07:14.183746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.183719    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-w6w4p"]
Oct 30 01:07:14.582054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.582021    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" path="/var/lib/kubelet/pods/14d96c09-fa3c-419a-8036-7189095b73e4/volumes"
Oct 30 01:07:14.582821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.582802    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" path="/var/lib/kubelet/pods/7f0b5121-f9ca-4203-9a44-ad3980d3f439/volumes"
Oct 30 01:07:14.583450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.583434    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" path="/var/lib/kubelet/pods/84b90433-33b5-49ee-9cd5-2faf0655badd/volumes"
Oct 30 01:07:14.584540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.584512    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" path="/var/lib/kubelet/pods/d00e8adc-8578-4335-86db-dd05f44ff6c4/volumes"
Oct 30 01:07:14.702077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.702035    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"]
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704091    5104 cpu_manager.go:414] "containerMap: removing container" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704120    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704138    5104 cpu_manager.go:414] "containerMap: removing container" podUID="691f9f06-7ff8-4031-ae6f-cb574e69c21b" containerName="revoke-keys"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704153    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691f9f06-7ff8-4031-ae6f-cb574e69c21b" containerName="revoke-keys"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704229    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704237    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704320    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704328    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704347    5104 cpu_manager.go:414] "containerMap: removing container" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704354    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704382    5104 cpu_manager.go:414] "containerMap: removing container" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704390    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.704840    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00e8adc-8578-4335-86db-dd05f44ff6c4" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.705075    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0b5121-f9ca-4203-9a44-ad3980d3f439" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.705100    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.705121    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d96c09-fa3c-419a-8036-7189095b73e4" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.705168    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b90433-33b5-49ee-9cd5-2faf0655badd" containerName="istio-proxy"
Oct 30 01:07:14.705972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.705187    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="691f9f06-7ff8-4031-ae6f-cb574e69c21b" containerName="revoke-keys"
Oct 30 01:07:14.713505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.713450    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:14.714138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.714113    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"]
Oct 30 01:07:14.726491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.726447    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:14.728383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.728359    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-262230\""
Oct 30 01:07:14.728592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.728577    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"]
Oct 30 01:07:14.741891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.741852    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"]
Oct 30 01:07:14.785522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.785490    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-262230-dockercfg-jvp6t\""
Oct 30 01:07:14.866495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.865647    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv97t\" (UniqueName: \"kubernetes.io/projected/fb63edbc-f6c8-4581-9537-f3b49f54165e-kube-api-access-sv97t\") pod \"payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:14.866495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.865790    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq28\" (UniqueName: \"kubernetes.io/projected/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-kube-api-access-qgq28\") pod \"payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:14.866495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.865935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:14.866495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.866097    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb63edbc-f6c8-4581-9537-f3b49f54165e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:14.970478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.970105    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:15.049159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:14.971149    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:15.049159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.047818    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb63edbc-f6c8-4581-9537-f3b49f54165e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:15.049159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.047982    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv97t\" (UniqueName: \"kubernetes.io/projected/fb63edbc-f6c8-4581-9537-f3b49f54165e-kube-api-access-sv97t\") pod \"payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:15.049159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.048127    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq28\" (UniqueName: \"kubernetes.io/projected/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-kube-api-access-qgq28\") pod \"payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:15.049159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.048735    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb63edbc-f6c8-4581-9537-f3b49f54165e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:15.188183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.188079    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq28\" (UniqueName: \"kubernetes.io/projected/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-kube-api-access-qgq28\") pod \"payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:15.210494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.210396    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv97t\" (UniqueName: \"kubernetes.io/projected/fb63edbc-f6c8-4581-9537-f3b49f54165e-kube-api-access-sv97t\") pod \"payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:15.351629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.351526    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:15.361814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:15.361720    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:16.549478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:16.547831    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 01:07:16.762333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:16.758689    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:07:16.762333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:16.759080    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:07:17.284817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:17.284781    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"]
Oct 30 01:07:17.322138 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:17.322104    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb63edbc_f6c8_4581_9537_f3b49f54165e.slice/crio-ad560debe903913fc761952c9e39d3bbc9aa170d930879c166c10aba67422f17 WatchSource:0}: Error finding container ad560debe903913fc761952c9e39d3bbc9aa170d930879c166c10aba67422f17: Status 404 returned error can't find the container with id ad560debe903913fc761952c9e39d3bbc9aa170d930879c166c10aba67422f17
Oct 30 01:07:17.542977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:17.542777    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:07:17.916941 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:17.916876    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a90c34_26ab_4eec_bab8_2c1c6a26bb76.slice/crio-d5ecead2a6b5c4b540ead558869b86b644a57ef480af793cc31b2b33f75204a3 WatchSource:0}: Error finding container d5ecead2a6b5c4b540ead558869b86b644a57ef480af793cc31b2b33f75204a3: Status 404 returned error can't find the container with id d5ecead2a6b5c4b540ead558869b86b644a57ef480af793cc31b2b33f75204a3
Oct 30 01:07:17.930450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:17.930417    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"]
Oct 30 01:07:17.995177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:17.992537    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" event={"ID":"fb63edbc-f6c8-4581-9537-f3b49f54165e","Type":"ContainerStarted","Data":"ad560debe903913fc761952c9e39d3bbc9aa170d930879c166c10aba67422f17"}
Oct 30 01:07:17.998603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:17.997544    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" event={"ID":"94a90c34-26ab-4eec-bab8-2c1c6a26bb76","Type":"ContainerStarted","Data":"d5ecead2a6b5c4b540ead558869b86b644a57ef480af793cc31b2b33f75204a3"}
Oct 30 01:07:19.049613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.049516    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" event={"ID":"fb63edbc-f6c8-4581-9537-f3b49f54165e","Type":"ContainerStarted","Data":"d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7"}
Oct 30 01:07:19.053378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.052556    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" event={"ID":"94a90c34-26ab-4eec-bab8-2c1c6a26bb76","Type":"ContainerStarted","Data":"b32ace3eb17a4329f02d287bf67e8f6ed033acbe7dbeb510664c5fe329560bf0"}
Oct 30 01:07:19.054108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.054089    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:19.054171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.054129    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:19.115377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.115322    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" podStartSLOduration=5.115306646 podStartE2EDuration="5.115306646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:19.112807142 +0000 UTC m=+1480.660497649" watchObservedRunningTime="2026-10-30 01:07:19.115306646 +0000 UTC m=+1480.662997143"
Oct 30 01:07:19.170775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.170723    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" podStartSLOduration=5.170708322 podStartE2EDuration="5.170708322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:19.169041417 +0000 UTC m=+1480.716731913" watchObservedRunningTime="2026-10-30 01:07:19.170708322 +0000 UTC m=+1480.718398819"
Oct 30 01:07:19.613584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.613545    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"]
Oct 30 01:07:19.619906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.619882    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:19.633681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.633617    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fb3370cb-dockercfg-vbp4x\""
Oct 30 01:07:19.633968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.633941    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb3370cb\""
Oct 30 01:07:19.635110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.635082    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"]
Oct 30 01:07:19.684501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.684421    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"]
Oct 30 01:07:19.752480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.732769    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cd6x\" (UniqueName: \"kubernetes.io/projected/5c20b9ac-bd1a-40de-a621-55c5ef126d96-kube-api-access-2cd6x\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:19.769885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.769851    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c20b9ac-bd1a-40de-a621-55c5ef126d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:19.811479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.805819    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:19.831487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.829169    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"]
Oct 30 01:07:19.876600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.876512    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bdbe07-8a8b-4ef4-81af-02f4e1423963-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:19.877015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.876989    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cd6x\" (UniqueName: \"kubernetes.io/projected/5c20b9ac-bd1a-40de-a621-55c5ef126d96-kube-api-access-2cd6x\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:19.877566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.877540    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkdpv\" (UniqueName: \"kubernetes.io/projected/02bdbe07-8a8b-4ef4-81af-02f4e1423963-kube-api-access-nkdpv\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:19.877782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.877765    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c20b9ac-bd1a-40de-a621-55c5ef126d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:19.898747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.898618    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c20b9ac-bd1a-40de-a621-55c5ef126d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:19.954608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:19.917505    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cd6x\" (UniqueName: \"kubernetes.io/projected/5c20b9ac-bd1a-40de-a621-55c5ef126d96-kube-api-access-2cd6x\") pod \"payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:20.003481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.003064    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bdbe07-8a8b-4ef4-81af-02f4e1423963-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:20.003481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.003232    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdpv\" (UniqueName: \"kubernetes.io/projected/02bdbe07-8a8b-4ef4-81af-02f4e1423963-kube-api-access-nkdpv\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:20.016828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.005446    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bdbe07-8a8b-4ef4-81af-02f4e1423963-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:20.068715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.068677    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdpv\" (UniqueName: \"kubernetes.io/projected/02bdbe07-8a8b-4ef4-81af-02f4e1423963-kube-api-access-nkdpv\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:20.233955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.233896    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:20.239450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.239365    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:20.557963 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:20.557928    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bdbe07_8a8b_4ef4_81af_02f4e1423963.slice/crio-0ad189c97aaf17c9f53bdd64bfa2cf5d3162ded07b3c51023f6da845072dc5b3 WatchSource:0}: Error finding container 0ad189c97aaf17c9f53bdd64bfa2cf5d3162ded07b3c51023f6da845072dc5b3: Status 404 returned error can't find the container with id 0ad189c97aaf17c9f53bdd64bfa2cf5d3162ded07b3c51023f6da845072dc5b3
Oct 30 01:07:20.559363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.559339    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"]
Oct 30 01:07:20.624139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:20.624108    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"]
Oct 30 01:07:21.181334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.181271    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h" event={"ID":"02bdbe07-8a8b-4ef4-81af-02f4e1423963","Type":"ContainerStarted","Data":"1185fca50950dda07e119af4bb5bc008922292ed8632a1ff8514b50470d6e2d3"}
Oct 30 01:07:21.181855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.181364    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h" event={"ID":"02bdbe07-8a8b-4ef4-81af-02f4e1423963","Type":"ContainerStarted","Data":"0ad189c97aaf17c9f53bdd64bfa2cf5d3162ded07b3c51023f6da845072dc5b3"}
Oct 30 01:07:21.207480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.201744    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:21.218909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.218759    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr" event={"ID":"5c20b9ac-bd1a-40de-a621-55c5ef126d96","Type":"ContainerStarted","Data":"48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07"}
Oct 30 01:07:21.218909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.218839    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:21.218909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.218869    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr" event={"ID":"5c20b9ac-bd1a-40de-a621-55c5ef126d96","Type":"ContainerStarted","Data":"a6c89e36d4189bad449274e43638ae165ae94988d6775920ed0acd7263d92440"}
Oct 30 01:07:21.251711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.251648    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h" podStartSLOduration=2.251632133 podStartE2EDuration="2.251632133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:21.247843054 +0000 UTC m=+1482.795533549" watchObservedRunningTime="2026-10-30 01:07:21.251632133 +0000 UTC m=+1482.799322629"
Oct 30 01:07:21.288856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.288801    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr" podStartSLOduration=2.2887858039999998 podStartE2EDuration="2.288785804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:21.283714923 +0000 UTC m=+1482.831405419" watchObservedRunningTime="2026-10-30 01:07:21.288785804 +0000 UTC m=+1482.836476299"
Oct 30 01:07:21.896479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.896424    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"]
Oct 30 01:07:21.908490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.908422    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:21.928544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.927326    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7e233f-openshift-default-dockercfg-b8njx\""
Oct 30 01:07:21.928544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:21.927682    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"]
Oct 30 01:07:22.047483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.043935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-certs\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.047483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.044097    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-credential-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.047483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.044185    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-envoy\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.072482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.061699    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4bw\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-kube-api-access-zw4bw\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.072482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.061803    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-data\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.101478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.098711    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1388f407-8532-47c6-b310-57e2c3302a07-istio-podinfo\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.101478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.098895    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.101478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.098941    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1388f407-8532-47c6-b310-57e2c3302a07-istiod-ca-cert\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.101478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.099000    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-istio-token\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.212481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.211743    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-certs\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.212481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212019    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-credential-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.212481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212159    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-envoy\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.212481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212375    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4bw\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-kube-api-access-zw4bw\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.212481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212490    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-data\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.213221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212542    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1388f407-8532-47c6-b310-57e2c3302a07-istio-podinfo\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.213221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212685    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.213221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212725    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1388f407-8532-47c6-b310-57e2c3302a07-istiod-ca-cert\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.213221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.212779    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-istio-token\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.225480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.213722    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-certs\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.225480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.213992    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-credential-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.246196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.243240    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-data\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.246196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.243623    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.246196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.244257    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1388f407-8532-47c6-b310-57e2c3302a07-istiod-ca-cert\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.267589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.256376    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-envoy\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.267589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.262856    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-istio-token\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.293682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.293648    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4bw\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-kube-api-access-zw4bw\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.312495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.301151    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1388f407-8532-47c6-b310-57e2c3302a07-istio-podinfo\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:22.559577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:22.558638    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:23.355017 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:23.354982    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1388f407_8532_47c6_b310_57e2c3302a07.slice/crio-e9bdb477faaeebec4a7d35595c62f5037c7f630e9acf31eca4466a4b30bd0835 WatchSource:0}: Error finding container e9bdb477faaeebec4a7d35595c62f5037c7f630e9acf31eca4466a4b30bd0835: Status 404 returned error can't find the container with id e9bdb477faaeebec4a7d35595c62f5037c7f630e9acf31eca4466a4b30bd0835
Oct 30 01:07:23.356095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:23.356073    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"]
Oct 30 01:07:23.358851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:23.358796    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:23.359111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:23.359087    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:23.359279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:23.359262    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:24.331365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:24.331323    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" event={"ID":"1388f407-8532-47c6-b310-57e2c3302a07","Type":"ContainerStarted","Data":"d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91"}
Oct 30 01:07:24.331569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:24.331393    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" event={"ID":"1388f407-8532-47c6-b310-57e2c3302a07","Type":"ContainerStarted","Data":"e9bdb477faaeebec4a7d35595c62f5037c7f630e9acf31eca4466a4b30bd0835"}
Oct 30 01:07:24.561916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:24.561878    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:24.634966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:24.634884    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:24.712497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:24.709603    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podStartSLOduration=3.709577505 podStartE2EDuration="3.709577505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:24.394986453 +0000 UTC m=+1485.942676949" watchObservedRunningTime="2026-10-30 01:07:24.709577505 +0000 UTC m=+1486.257268006"
Oct 30 01:07:25.342341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:25.342307    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:25.343362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:25.343336    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:07:28.621649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.621606    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"]
Oct 30 01:07:28.626763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.626736    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:28.646437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.646405    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"]
Oct 30 01:07:28.651778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.651738    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7e233f\""
Oct 30 01:07:28.651935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.651839    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7e233f-dockercfg-fcmsm\""
Oct 30 01:07:28.843479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.821052    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4l8\" (UniqueName: \"kubernetes.io/projected/a8f0a3cf-9833-409e-a859-de471cc262fb-kube-api-access-mk4l8\") pod \"payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:28.864002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.863963    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f0a3cf-9833-409e-a859-de471cc262fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:28.996999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.921742    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"]
Oct 30 01:07:28.996999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.986380    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4l8\" (UniqueName: \"kubernetes.io/projected/a8f0a3cf-9833-409e-a859-de471cc262fb-kube-api-access-mk4l8\") pod \"payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:28.996999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:28.986536    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f0a3cf-9833-409e-a859-de471cc262fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:29.027862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.027827    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f0a3cf-9833-409e-a859-de471cc262fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:29.047480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.032113    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"]
Oct 30 01:07:29.047480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.032238    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.066139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.066102    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4l8\" (UniqueName: \"kubernetes.io/projected/a8f0a3cf-9833-409e-a859-de471cc262fb-kube-api-access-mk4l8\") pod \"payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:29.214429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.214383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvv4v\" (UniqueName: \"kubernetes.io/projected/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-kube-api-access-nvv4v\") pod \"payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.241050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.241008    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.243488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.243430    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:29.345471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.345419    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvv4v\" (UniqueName: \"kubernetes.io/projected/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-kube-api-access-nvv4v\") pod \"payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.345669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.345654    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.350119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.350088    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.365614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.365577    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvv4v\" (UniqueName: \"kubernetes.io/projected/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-kube-api-access-nvv4v\") pod \"payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.452999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.452971    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:29.576851 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:29.575837    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f0a3cf_9833_409e_a859_de471cc262fb.slice/crio-5e935def89b00add9dec4d9db78085894ef6c38d498efc925db1ea18d724bf8c WatchSource:0}: Error finding container 5e935def89b00add9dec4d9db78085894ef6c38d498efc925db1ea18d724bf8c: Status 404 returned error can't find the container with id 5e935def89b00add9dec4d9db78085894ef6c38d498efc925db1ea18d724bf8c
Oct 30 01:07:29.580271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.580244    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"]
Oct 30 01:07:29.790446 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:29.790414    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2fe939_5dcb_49ca_b8d3_4ef3834e83c7.slice/crio-ef7155ece4596a78ca487edafdd3506d2a01db43dac0728a35c4c9650ef82d5f WatchSource:0}: Error finding container ef7155ece4596a78ca487edafdd3506d2a01db43dac0728a35c4c9650ef82d5f: Status 404 returned error can't find the container with id ef7155ece4596a78ca487edafdd3506d2a01db43dac0728a35c4c9650ef82d5f
Oct 30 01:07:29.792381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:29.792347    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"]
Oct 30 01:07:30.562189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.561962    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls" event={"ID":"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7","Type":"ContainerStarted","Data":"5a366c179f72578357d2ba7d83d9ea8ce9213c736b8dbe9ff41861e84657ece0"}
Oct 30 01:07:30.562189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.562031    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls" event={"ID":"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7","Type":"ContainerStarted","Data":"ef7155ece4596a78ca487edafdd3506d2a01db43dac0728a35c4c9650ef82d5f"}
Oct 30 01:07:30.562189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.562160    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:30.567014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.566987    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" event={"ID":"a8f0a3cf-9833-409e-a859-de471cc262fb","Type":"ContainerStarted","Data":"44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674"}
Oct 30 01:07:30.567876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.567851    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:30.567990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.567885    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" event={"ID":"a8f0a3cf-9833-409e-a859-de471cc262fb","Type":"ContainerStarted","Data":"5e935def89b00add9dec4d9db78085894ef6c38d498efc925db1ea18d724bf8c"}
Oct 30 01:07:30.604518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.604193    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls" podStartSLOduration=2.604177362 podStartE2EDuration="2.604177362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:30.599990416 +0000 UTC m=+1492.147680911" watchObservedRunningTime="2026-10-30 01:07:30.604177362 +0000 UTC m=+1492.151867857"
Oct 30 01:07:30.638111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:30.638049    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" podStartSLOduration=2.63803296 podStartE2EDuration="2.63803296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:30.635607261 +0000 UTC m=+1492.183297768" watchObservedRunningTime="2026-10-30 01:07:30.63803296 +0000 UTC m=+1492.185723456"
Oct 30 01:07:31.224402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:31.224368    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:07:31.224956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:31.224495    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:07:32.292861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:32.292830    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:07:33.287503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:33.287472    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:07:37.142002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.141967    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"]
Oct 30 01:07:37.142501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.142187    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy" containerID="cri-o://1d133be20ad9c919c5fa34de9e4e31708ce248f818cdac60caf97e1c688592b0" gracePeriod=30
Oct 30 01:07:37.152346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.152309    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"]
Oct 30 01:07:37.152654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.152610    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy" containerID="cri-o://04a942d802fe5764d9663d2058afea17e59e7026c7fea203ba1b500b7683632e" gracePeriod=30
Oct 30 01:07:37.158435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.158405    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"]
Oct 30 01:07:37.158640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.158616    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy" containerID="cri-o://4798382ae5e62f842ca4becc398b9deff0d4c9094c6c8f9d41099aacd363605b" gracePeriod=30
Oct 30 01:07:37.445108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.445056    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"]
Oct 30 01:07:37.445770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:37.445727    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy" containerID="cri-o://53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b" gracePeriod=30
Oct 30 01:07:38.860136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:38.860105    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:07:38.866281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:38.866254    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:07:38.867799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:38.867777    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:07:38.883019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:38.882980    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:07:39.232529 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:07:39.230663    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:07:41.582054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.582025    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:07:41.582607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.582100    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:07:41.662106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.662068    5104 scope.go:122] "RemoveContainer" containerID="16b847c7233baab699fefc2ba67e9ceb4c4e5a88710733b8e3c7a9915b8ae954"
Oct 30 01:07:41.685434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.682260    5104 scope.go:122] "RemoveContainer" containerID="bf147eea7ff16949b155df573477ee7dd0a6b543542700f0738525a8cda144f6"
Oct 30 01:07:41.708134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.708103    5104 scope.go:122] "RemoveContainer" containerID="a3fd3c2bdad43e1caf870ae4fabec32d89dc9e309f0319b1249147ce4eb98895"
Oct 30 01:07:41.734333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.734302    5104 scope.go:122] "RemoveContainer" containerID="6fcf931b15c0154d7104038fbd9ec87d22804518ba3aa41ffd6f513cbaa2ff80"
Oct 30 01:07:41.751291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.751261    5104 scope.go:122] "RemoveContainer" containerID="1d08a4f9f4cc17b7317a9b5bace93e60d7051a8338ec3d5d3c6cc3f8629c1830"
Oct 30 01:07:41.762676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.762654    5104 scope.go:122] "RemoveContainer" containerID="4be6d78ded1528d17a946da07d374205e4d6674d1009329b5133295c0c34544f"
Oct 30 01:07:41.774572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.774548    5104 scope.go:122] "RemoveContainer" containerID="f1fc7709afe9c71f3773e3768b13511ab3908dd280d62d813c6bdbf77699e7e4"
Oct 30 01:07:41.800768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.789435    5104 scope.go:122] "RemoveContainer" containerID="a9b5b973f695f7a2bdf137674fa3a4e1fc582b27769a36f904d36ca480f8d45b"
Oct 30 01:07:41.970278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:41.970246    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"]
Oct 30 01:07:42.031492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.012238    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.088741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.088710    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:42.094002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.089571    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:07:42.094858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.094828    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.095238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.095215    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.095717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.095696    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6jv\" (UniqueName: \"kubernetes.io/projected/c25c364f-5c34-4e57-8352-d63c08d49931-kube-api-access-6m6jv\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.132104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.130740    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:07:42.132104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.130904    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:07:42.160599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.158648    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"]
Oct 30 01:07:42.204995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.203951    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6jv\" (UniqueName: \"kubernetes.io/projected/c25c364f-5c34-4e57-8352-d63c08d49931-kube-api-access-6m6jv\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.204995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.204082    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.204995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.204212    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.220415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.220317    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.222478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.222400    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.228981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.227669    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6jv\" (UniqueName: \"kubernetes.io/projected/c25c364f-5c34-4e57-8352-d63c08d49931-kube-api-access-6m6jv\") pod \"maas-api-revoke-sub-4294283d-c588f\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.233211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.233179    5104 scope.go:122] "RemoveContainer" containerID="fecb6bd902ae6017e72ab2b175198635770dd4c0511a5a9a3d9816d26a0aa1d6"
Oct 30 01:07:42.432241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.425782    5104 scope.go:122] "RemoveContainer" containerID="c29633b0dd7b301e394330f34ae73909daceec236b3ec8cddd8832eeedbd84a7"
Oct 30 01:07:42.446066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.446040    5104 scope.go:122] "RemoveContainer" containerID="84ffc2d62c400a1c40664c33675ca73202807161895706f8162a17accabd20f6"
Oct 30 01:07:42.468397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.468113    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:42.472752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.472646    5104 scope.go:122] "RemoveContainer" containerID="33d9c6781bbdae9a8d5c786fc0d20074a4c88f61d3a8f7b77b41e5b381b23560"
Oct 30 01:07:42.514097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.509816    5104 scope.go:122] "RemoveContainer" containerID="a351bf1ee3e3104b3b819e715f2fc6fb615db6c4cf8e9f57f404efab7225d356"
Oct 30 01:07:42.721009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.716811    5104 scope.go:122] "RemoveContainer" containerID="706bffa71cca3d9d7240550567353b9637e03f762f3ece069caf95bcec417a9e"
Oct 30 01:07:42.795569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.779912    5104 scope.go:122] "RemoveContainer" containerID="d40ec4da66a41953bbad1654716cee894cac82a2fef2d5608700a7f0b1b3d1f1"
Oct 30 01:07:42.802104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.802073    5104 generic.go:356] "Generic (PLEG): container finished" podID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerID="4798382ae5e62f842ca4becc398b9deff0d4c9094c6c8f9d41099aacd363605b" exitCode=0
Oct 30 01:07:42.802212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.802175    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" event={"ID":"79cfbc0c-77b6-4a20-97b5-e53787626b3a","Type":"ContainerDied","Data":"4798382ae5e62f842ca4becc398b9deff0d4c9094c6c8f9d41099aacd363605b"}
Oct 30 01:07:42.817728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.817507    5104 scope.go:122] "RemoveContainer" containerID="821d5ecd67198fa10c43985a980156098157562184d44478521539c3eacc5c5c"
Oct 30 01:07:42.834364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.833029    5104 generic.go:356] "Generic (PLEG): container finished" podID="5d935bfd-813c-41f8-9721-b052f383076f" containerID="1d133be20ad9c919c5fa34de9e4e31708ce248f818cdac60caf97e1c688592b0" exitCode=0
Oct 30 01:07:42.834364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.833124    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" event={"ID":"5d935bfd-813c-41f8-9721-b052f383076f","Type":"ContainerDied","Data":"1d133be20ad9c919c5fa34de9e4e31708ce248f818cdac60caf97e1c688592b0"}
Oct 30 01:07:42.835244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.835216    5104 generic.go:356] "Generic (PLEG): container finished" podID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerID="04a942d802fe5764d9663d2058afea17e59e7026c7fea203ba1b500b7683632e" exitCode=0
Oct 30 01:07:42.835311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.835294    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" event={"ID":"3316e4b0-dc86-4cd1-9974-686040ba399f","Type":"ContainerDied","Data":"04a942d802fe5764d9663d2058afea17e59e7026c7fea203ba1b500b7683632e"}
Oct 30 01:07:42.853205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.853174    5104 scope.go:122] "RemoveContainer" containerID="2bde5c094a688580a5450e4e6e247447177fee63488e56e288e11e7b489d3105"
Oct 30 01:07:42.859212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.859176    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 01:07:42.868936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.868907    5104 scope.go:122] "RemoveContainer" containerID="08cc8b01e3145d361d6e341252cc20611074b7f8d2dfdcb02c1b28d1901c9295"
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.942742    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-data\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.942858    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-envoy\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.942891    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-kube-api-access-7h6rq\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-kube-api-access-7h6rq\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.942969    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-podinfo\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.943004    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-socket\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.943151    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istiod-ca-cert\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.943236    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-credential-socket\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.943283    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-token\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.943359    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-certs\") pod \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\" (UID: \"79cfbc0c-77b6-4a20-97b5-e53787626b3a\") "
Oct 30 01:07:42.945478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.945289    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-certs" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:42.954987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.950388    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-socket" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:42.954987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.953522    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-envoy" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:42.954987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.954533    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istiod-ca-cert" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:42.957981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.957737    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-credential-socket" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:42.957981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.957802    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-token" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:42.960512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.958991    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-kube-api-access-7h6rq" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "kube-api-access-7h6rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:42.967596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.960922    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-podinfo" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:42.967993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:42.967969    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-data" pod "79cfbc0c-77b6-4a20-97b5-e53787626b3a" (UID: "79cfbc0c-77b6-4a20-97b5-e53787626b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.042694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.041683    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:07:43.044383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.044358    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:07:43.045009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.044988    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-credential-socket\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.045094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045042    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-token\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-token\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.045157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045124    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-envoy\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.045216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045197    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-certs\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045303    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-credential-socket" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045435    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-podinfo\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045496    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3316e4b0-dc86-4cd1-9974-686040ba399f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316e4b0-dc86-4cd1-9974-686040ba399f-istiod-ca-cert\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045574    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-certs" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045675    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-kube-api-access-d7xdl\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-kube-api-access-d7xdl\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045832    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-data\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.045909    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-socket\") pod \"3316e4b0-dc86-4cd1-9974-686040ba399f\" (UID: \"3316e4b0-dc86-4cd1-9974-686040ba399f\") "
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.046524    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-data" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.046856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.046727    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-socket" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.047446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047416    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3316e4b0-dc86-4cd1-9974-686040ba399f-istiod-ca-cert" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:43.047704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047668    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.047794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047710    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.047794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047727    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.047794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047743    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.047794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047757    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.047794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047771    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047809    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6rq\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-kube-api-access-7h6rq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047823    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047838    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047851    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047866    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047880    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316e4b0-dc86-4cd1-9974-686040ba399f-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047894    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cfbc0c-77b6-4a20-97b5-e53787626b3a-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.047908    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cfbc0c-77b6-4a20-97b5-e53787626b3a-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.048876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.048851    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-envoy" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.051194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.051169    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-kube-api-access-d7xdl" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "kube-api-access-d7xdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:43.051913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.051889    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-podinfo" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:43.054560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.054536    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-token" pod "3316e4b0-dc86-4cd1-9974-686040ba399f" (UID: "3316e4b0-dc86-4cd1-9974-686040ba399f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:43.141167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.140650    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:07:43.147385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.147161    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"]
Oct 30 01:07:43.150962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.150017    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-credential-socket\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.150962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.150820    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-kube-api-access-d5txl\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-kube-api-access-d5txl\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.150962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.150874    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-data\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.150962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.150906    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-podinfo\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.151210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151007    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-kube-api-access-w9r7x\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-kube-api-access-w9r7x\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.151210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151064    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d935bfd-813c-41f8-9721-b052f383076f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d935bfd-813c-41f8-9721-b052f383076f-istiod-ca-cert\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.151210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151109    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-credential-socket\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.151210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151177    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-envoy\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.151502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151258    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-data\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.151502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151323    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381b806e-98cd-499f-8f1d-e6166a3956d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381b806e-98cd-499f-8f1d-e6166a3956d6-istiod-ca-cert\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.151502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151406    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d935bfd-813c-41f8-9721-b052f383076f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d935bfd-813c-41f8-9721-b052f383076f-istio-podinfo\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.151614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151522    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-certs\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.151614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151586    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-envoy\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.151705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.151630    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-token\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-token\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.152168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.152142    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-istio-token\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-istio-token\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.152262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.152245    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-socket\") pod \"5d935bfd-813c-41f8-9721-b052f383076f\" (UID: \"5d935bfd-813c-41f8-9721-b052f383076f\") "
Oct 30 01:07:43.152302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.152283    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-socket\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.152337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.152325    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-certs\") pod \"381b806e-98cd-499f-8f1d-e6166a3956d6\" (UID: \"381b806e-98cd-499f-8f1d-e6166a3956d6\") "
Oct 30 01:07:43.153396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.153364    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7xdl\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-kube-api-access-d7xdl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.153553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.153398    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.153553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.153413    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.153553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.153426    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316e4b0-dc86-4cd1-9974-686040ba399f-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.153724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.153688    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-certs" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.153889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.153864    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-data" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.154052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.154037    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-credential-socket" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.154109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.154074    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b806e-98cd-499f-8f1d-e6166a3956d6-istiod-ca-cert" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:43.155558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.155384    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-socket" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.156030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.155996    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-socket" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.156215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.156191    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-certs" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.156338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.156313    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-credential-socket" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.156691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.156652    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d935bfd-813c-41f8-9721-b052f383076f-istiod-ca-cert" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:43.157053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.157022    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-data" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.158021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.157832    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-token" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:43.159927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.159851    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-envoy" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.160209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.160043    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-kube-api-access-w9r7x" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "kube-api-access-w9r7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:43.162481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.160543    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-istio-token" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:43.165391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.165311    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-envoy" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:43.165741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.165713    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-kube-api-access-d5txl" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "kube-api-access-d5txl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:43.166248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.166157    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d935bfd-813c-41f8-9721-b052f383076f-istio-podinfo" pod "5d935bfd-813c-41f8-9721-b052f383076f" (UID: "5d935bfd-813c-41f8-9721-b052f383076f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:43.166248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.166221    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-podinfo" pod "381b806e-98cd-499f-8f1d-e6166a3956d6" (UID: "381b806e-98cd-499f-8f1d-e6166a3956d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:43.255340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255306    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9r7x\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-kube-api-access-w9r7x\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255352    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d935bfd-813c-41f8-9721-b052f383076f-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255372    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255387    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255401    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255415    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381b806e-98cd-499f-8f1d-e6166a3956d6-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255430    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d935bfd-813c-41f8-9721-b052f383076f-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255444    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255504    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255518    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255532    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d935bfd-813c-41f8-9721-b052f383076f-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255545    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255560    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255576    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255592    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381b806e-98cd-499f-8f1d-e6166a3956d6-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255605    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5txl\" (UniqueName: \"kubernetes.io/projected/381b806e-98cd-499f-8f1d-e6166a3956d6-kube-api-access-d5txl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255620    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d935bfd-813c-41f8-9721-b052f383076f-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.255642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.255634    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381b806e-98cd-499f-8f1d-e6166a3956d6-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.832231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.832195    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"]
Oct 30 01:07:43.833257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833230    5104 cpu_manager.go:414] "containerMap: removing container" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy"
Oct 30 01:07:43.833257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833256    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy"
Oct 30 01:07:43.833428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833290    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy"
Oct 30 01:07:43.833428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833297    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy"
Oct 30 01:07:43.833428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833317    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy"
Oct 30 01:07:43.833428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833325    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy"
Oct 30 01:07:43.833428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833419    5104 cpu_manager.go:414] "containerMap: removing container" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy"
Oct 30 01:07:43.833689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833429    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy"
Oct 30 01:07:43.833689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833682    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" containerName="istio-proxy"
Oct 30 01:07:43.833789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833700    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerName="istio-proxy"
Oct 30 01:07:43.833789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833715    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" containerName="istio-proxy"
Oct 30 01:07:43.833789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.833727    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d935bfd-813c-41f8-9721-b052f383076f" containerName="istio-proxy"
Oct 30 01:07:43.838321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.838298    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.850047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.850021    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"
Oct 30 01:07:43.850047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.850028    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr" event={"ID":"5d935bfd-813c-41f8-9721-b052f383076f","Type":"ContainerDied","Data":"171772511ce1a42f3d83850410d7393abb0a0ce9fa8aa7a0c8339ce3fcf4c79b"}
Oct 30 01:07:43.850258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.850092    5104 scope.go:122] "RemoveContainer" containerID="1d133be20ad9c919c5fa34de9e4e31708ce248f818cdac60caf97e1c688592b0"
Oct 30 01:07:43.856748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.856715    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"
Oct 30 01:07:43.858935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.858908    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb" event={"ID":"3316e4b0-dc86-4cd1-9974-686040ba399f","Type":"ContainerDied","Data":"f90c6f64737f3911ef2be46952f3e4bd74b58dbd1a7ff2009516e08a1d4c204e"}
Oct 30 01:07:43.861670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.860360    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"]
Oct 30 01:07:43.865555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.865529    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.865672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.865606    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.865672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.865659    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.865803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.865728    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.865803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.865783    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57324b74-cc8c-4b87-8bb7-fa51af899929-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.865917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.865821    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.865968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.865950    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5vs\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-kube-api-access-xb5vs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.866207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.866191    5104 scope.go:122] "RemoveContainer" containerID="04a942d802fe5764d9663d2058afea17e59e7026c7fea203ba1b500b7683632e"
Oct 30 01:07:43.871106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.871080    5104 generic.go:356] "Generic (PLEG): container finished" podID="381b806e-98cd-499f-8f1d-e6166a3956d6" containerID="53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b" exitCode=0
Oct 30 01:07:43.871211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.871178    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" event={"ID":"381b806e-98cd-499f-8f1d-e6166a3956d6","Type":"ContainerDied","Data":"53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b"}
Oct 30 01:07:43.871274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.871246    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5" event={"ID":"381b806e-98cd-499f-8f1d-e6166a3956d6","Type":"ContainerDied","Data":"a1acdd6df34df0f8355db4091d49f683a36e357b0bcc60dea2cba364111387ec"}
Oct 30 01:07:43.871367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.871348    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"
Oct 30 01:07:43.887804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.887776    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f" event={"ID":"c25c364f-5c34-4e57-8352-d63c08d49931","Type":"ContainerStarted","Data":"82875eed34f15d7bbc989b7f8ba7d893791258c62902f337bb7644c41eca8146"}
Oct 30 01:07:43.888563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.888535    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f" event={"ID":"c25c364f-5c34-4e57-8352-d63c08d49931","Type":"ContainerStarted","Data":"ba9e74343c6d2b11280a0785df7b76c5052e2741bdfee31b678d869a59d3209b"}
Oct 30 01:07:43.890882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.890857    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd" event={"ID":"79cfbc0c-77b6-4a20-97b5-e53787626b3a","Type":"ContainerDied","Data":"5b3403b90b9bc9cebc926cfbff9d3232c5aeb5e65fdff8c307312180bd5f3dc9"}
Oct 30 01:07:43.890974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.890864    5104 scope.go:122] "RemoveContainer" containerID="53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b"
Oct 30 01:07:43.890974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.890941    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"
Oct 30 01:07:43.909785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.906120    5104 scope.go:122] "RemoveContainer" containerID="53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b"
Oct 30 01:07:43.909785 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:07:43.906556    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b\": container with ID starting with 53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b not found: ID does not exist" containerID="53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b"
Oct 30 01:07:43.909785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.906595    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b"} err="failed to get container status \"53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b\": rpc error: code = NotFound desc = could not find container \"53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b\": container with ID starting with 53f80de35c67d89c8c3f57358c1efa2da9fe2c9f07f907bd56022882b5c47f5b not found: ID does not exist"
Oct 30 01:07:43.909785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.906619    5104 scope.go:122] "RemoveContainer" containerID="4798382ae5e62f842ca4becc398b9deff0d4c9094c6c8f9d41099aacd363605b"
Oct 30 01:07:43.920565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.920514    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"]
Oct 30 01:07:43.929050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.929022    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-bqjqb"]
Oct 30 01:07:43.952246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.952166    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"]
Oct 30 01:07:43.958116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.958085    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-xjdpr"]
Oct 30 01:07:43.966971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.966937    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5vs\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-kube-api-access-xb5vs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.967122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.967003    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.967122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.967070    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.967254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.967145    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.967254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.967247    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.967358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.967335    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57324b74-cc8c-4b87-8bb7-fa51af899929-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.967441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.967415    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.968086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.968060    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.969890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.969866    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57324b74-cc8c-4b87-8bb7-fa51af899929-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.969984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.969958    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.970053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.970036    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.970368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.970346    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.985299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.985268    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:43.985953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:43.985925    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5vs\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-kube-api-access-xb5vs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6c2h\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:44.004317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.004232    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f" podStartSLOduration=3.004217591 podStartE2EDuration="3.004217591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:44.002450377 +0000 UTC m=+1505.550140872" watchObservedRunningTime="2026-10-30 01:07:44.004217591 +0000 UTC m=+1505.551908086"
Oct 30 01:07:44.039882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.039845    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"]
Oct 30 01:07:44.045077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.045042    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-9qftd"]
Oct 30 01:07:44.078551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.078512    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"]
Oct 30 01:07:44.084215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.084183    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-wlrz5"]
Oct 30 01:07:44.156623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.156588    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:44.487436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.487403    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"]
Oct 30 01:07:44.488881 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:07:44.488833    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57324b74_cc8c_4b87_8bb7_fa51af899929.slice/crio-8deee88d3a3508ea00132df7fd3f2ed24c1a0372a8f2d87519aa092564d0cc8e WatchSource:0}: Error finding container 8deee88d3a3508ea00132df7fd3f2ed24c1a0372a8f2d87519aa092564d0cc8e: Status 404 returned error can't find the container with id 8deee88d3a3508ea00132df7fd3f2ed24c1a0372a8f2d87519aa092564d0cc8e
Oct 30 01:07:44.492055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.492023    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:44.492155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.492080    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:07:44.583914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.583868    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3316e4b0-dc86-4cd1-9974-686040ba399f" path="/var/lib/kubelet/pods/3316e4b0-dc86-4cd1-9974-686040ba399f/volumes"
Oct 30 01:07:44.584656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.584638    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381b806e-98cd-499f-8f1d-e6166a3956d6" path="/var/lib/kubelet/pods/381b806e-98cd-499f-8f1d-e6166a3956d6/volumes"
Oct 30 01:07:44.585306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.585290    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d935bfd-813c-41f8-9721-b052f383076f" path="/var/lib/kubelet/pods/5d935bfd-813c-41f8-9721-b052f383076f/volumes"
Oct 30 01:07:44.586294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.586276    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cfbc0c-77b6-4a20-97b5-e53787626b3a" path="/var/lib/kubelet/pods/79cfbc0c-77b6-4a20-97b5-e53787626b3a/volumes"
Oct 30 01:07:44.926909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.926868    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h" event={"ID":"57324b74-cc8c-4b87-8bb7-fa51af899929","Type":"ContainerStarted","Data":"ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f"}
Oct 30 01:07:44.927545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.927521    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h" event={"ID":"57324b74-cc8c-4b87-8bb7-fa51af899929","Type":"ContainerStarted","Data":"8deee88d3a3508ea00132df7fd3f2ed24c1a0372a8f2d87519aa092564d0cc8e"}
Oct 30 01:07:44.927945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.927922    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:44.934124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.934098    5104 generic.go:356] "Generic (PLEG): container finished" podID="c25c364f-5c34-4e57-8352-d63c08d49931" containerID="82875eed34f15d7bbc989b7f8ba7d893791258c62902f337bb7644c41eca8146" exitCode=0
Oct 30 01:07:44.934247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.934183    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f" event={"ID":"c25c364f-5c34-4e57-8352-d63c08d49931","Type":"ContainerDied","Data":"82875eed34f15d7bbc989b7f8ba7d893791258c62902f337bb7644c41eca8146"}
Oct 30 01:07:44.971709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:44.971651    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h" podStartSLOduration=1.971637597 podStartE2EDuration="1.971637597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:44.969888437 +0000 UTC m=+1506.517578934" watchObservedRunningTime="2026-10-30 01:07:44.971637597 +0000 UTC m=+1506.519328092"
Oct 30 01:07:45.952158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:45.952129    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:07:46.421845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.421818    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:46.441887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.441852    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-xks-ca\") pod \"c25c364f-5c34-4e57-8352-d63c08d49931\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") "
Oct 30 01:07:46.442075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.441964    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25c364f-5c34-4e57-8352-d63c08d49931-kube-api-access-6m6jv\" (UniqueName: \"kubernetes.io/projected/c25c364f-5c34-4e57-8352-d63c08d49931-kube-api-access-6m6jv\") pod \"c25c364f-5c34-4e57-8352-d63c08d49931\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") "
Oct 30 01:07:46.442075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.442043    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-openshift-ca\") pod \"c25c364f-5c34-4e57-8352-d63c08d49931\" (UID: \"c25c364f-5c34-4e57-8352-d63c08d49931\") "
Oct 30 01:07:46.442495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.442451    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-openshift-ca" pod "c25c364f-5c34-4e57-8352-d63c08d49931" (UID: "c25c364f-5c34-4e57-8352-d63c08d49931"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:46.442942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.442922    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:46.444067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.444041    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-xks-ca" pod "c25c364f-5c34-4e57-8352-d63c08d49931" (UID: "c25c364f-5c34-4e57-8352-d63c08d49931"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:46.444284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.444268    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25c364f-5c34-4e57-8352-d63c08d49931-kube-api-access-6m6jv" pod "c25c364f-5c34-4e57-8352-d63c08d49931" (UID: "c25c364f-5c34-4e57-8352-d63c08d49931"). InnerVolumeSpecName "kube-api-access-6m6jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:46.544257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.544218    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25c364f-5c34-4e57-8352-d63c08d49931-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:46.544257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.544253    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6jv\" (UniqueName: \"kubernetes.io/projected/c25c364f-5c34-4e57-8352-d63c08d49931-kube-api-access-6m6jv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:07:46.959830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.959799    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"
Oct 30 01:07:46.960228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.959830    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f" event={"ID":"c25c364f-5c34-4e57-8352-d63c08d49931","Type":"ContainerDied","Data":"ba9e74343c6d2b11280a0785df7b76c5052e2741bdfee31b678d869a59d3209b"}
Oct 30 01:07:46.960228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:46.959884    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9e74343c6d2b11280a0785df7b76c5052e2741bdfee31b678d869a59d3209b"
Oct 30 01:07:50.703161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:07:50.703120    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:08:01.032543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:01.032509    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"]
Oct 30 01:08:01.044653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:01.044609    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283a11e-59spf"]
Oct 30 01:08:02.563481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.560508    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"]
Oct 30 01:08:02.563481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.561835    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c25c364f-5c34-4e57-8352-d63c08d49931" containerName="revoke-keys"
Oct 30 01:08:02.563481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.561858    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25c364f-5c34-4e57-8352-d63c08d49931" containerName="revoke-keys"
Oct 30 01:08:02.563481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.562111    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25c364f-5c34-4e57-8352-d63c08d49931" containerName="revoke-keys"
Oct 30 01:08:02.582943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.581909    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"]
Oct 30 01:08:02.608539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.605780    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:02.621488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.620471    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:02.642843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.641401    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:02.686019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.685603    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg59w\" (UniqueName: \"kubernetes.io/projected/fb1b67db-8039-4846-9897-f2f46846171e-kube-api-access-sg59w\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:02.686019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.685709    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:02.880626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.847997    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:02.880626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.848069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg59w\" (UniqueName: \"kubernetes.io/projected/fb1b67db-8039-4846-9897-f2f46846171e-kube-api-access-sg59w\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:02.880626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.848130    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:02.929549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.913227    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:08:02.929765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.918241    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:08:02.929893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:02.920340    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:08:03.152529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:03.049545    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:03.152529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:03.070529    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:03.152529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:03.072578    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50167c9-f5fe-45b9-83b1-8f495bfc65c0" path="/var/lib/kubelet/pods/a50167c9-f5fe-45b9-83b1-8f495bfc65c0/volumes"
Oct 30 01:08:03.348550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:03.320932    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg59w\" (UniqueName: \"kubernetes.io/projected/fb1b67db-8039-4846-9897-f2f46846171e-kube-api-access-sg59w\") pod \"maas-api-revoke-keys-e2e-api-a-260268-nnfkt\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:03.373483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:03.351216    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"]
Oct 30 01:08:03.423903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:03.423848    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d855-79snv"]
Oct 30 01:08:03.527480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:03.522126    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:04.023482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:04.021538    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"]
Oct 30 01:08:04.638949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:04.638916    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691f9f06-7ff8-4031-ae6f-cb574e69c21b" path="/var/lib/kubelet/pods/691f9f06-7ff8-4031-ae6f-cb574e69c21b/volumes"
Oct 30 01:08:04.639555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:04.639534    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt" event={"ID":"fb1b67db-8039-4846-9897-f2f46846171e","Type":"ContainerStarted","Data":"e8860e6dda2ce83852cb2c4e74a1db57c91e553cdfdc0ecc2953e945cd31884b"}
Oct 30 01:08:05.626491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:05.626433    5104 generic.go:356] "Generic (PLEG): container finished" podID="fb1b67db-8039-4846-9897-f2f46846171e" containerID="23f829229dd55c87d8c33831d15bf1194b59fa03b23faf890da3a513baf1b768" exitCode=0
Oct 30 01:08:05.627951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:05.627572    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt" event={"ID":"fb1b67db-8039-4846-9897-f2f46846171e","Type":"ContainerDied","Data":"23f829229dd55c87d8c33831d15bf1194b59fa03b23faf890da3a513baf1b768"}
Oct 30 01:08:06.881817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:06.881691    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:07.001959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.001910    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-xks-ca\") pod \"fb1b67db-8039-4846-9897-f2f46846171e\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") "
Oct 30 01:08:07.002177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.002034    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-openshift-ca\") pod \"fb1b67db-8039-4846-9897-f2f46846171e\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") "
Oct 30 01:08:07.002269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.002230    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1b67db-8039-4846-9897-f2f46846171e-kube-api-access-sg59w\" (UniqueName: \"kubernetes.io/projected/fb1b67db-8039-4846-9897-f2f46846171e-kube-api-access-sg59w\") pod \"fb1b67db-8039-4846-9897-f2f46846171e\" (UID: \"fb1b67db-8039-4846-9897-f2f46846171e\") "
Oct 30 01:08:07.004363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.004330    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-openshift-ca" pod "fb1b67db-8039-4846-9897-f2f46846171e" (UID: "fb1b67db-8039-4846-9897-f2f46846171e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:07.006108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.006076    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1b67db-8039-4846-9897-f2f46846171e-kube-api-access-sg59w" pod "fb1b67db-8039-4846-9897-f2f46846171e" (UID: "fb1b67db-8039-4846-9897-f2f46846171e"). InnerVolumeSpecName "kube-api-access-sg59w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:07.007336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.007303    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-xks-ca" pod "fb1b67db-8039-4846-9897-f2f46846171e" (UID: "fb1b67db-8039-4846-9897-f2f46846171e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:07.106446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.106407    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:07.106446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.106446    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg59w\" (UniqueName: \"kubernetes.io/projected/fb1b67db-8039-4846-9897-f2f46846171e-kube-api-access-sg59w\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:07.106722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.106485    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1b67db-8039-4846-9897-f2f46846171e-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:07.650441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.650410    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"
Oct 30 01:08:07.650441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.650442    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt" event={"ID":"fb1b67db-8039-4846-9897-f2f46846171e","Type":"ContainerDied","Data":"e8860e6dda2ce83852cb2c4e74a1db57c91e553cdfdc0ecc2953e945cd31884b"}
Oct 30 01:08:07.650708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:07.650509    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8860e6dda2ce83852cb2c4e74a1db57c91e553cdfdc0ecc2953e945cd31884b"
Oct 30 01:08:09.112094 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:08:09.111993    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:08:09.670934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.670901    5104 generic.go:356] "Generic (PLEG): container finished" podID="fc10b078-6065-4c3b-9655-306808b43f55" containerID="93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe" exitCode=0
Oct 30 01:08:09.671141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.670981    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" event={"ID":"fc10b078-6065-4c3b-9655-306808b43f55","Type":"ContainerDied","Data":"93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe"}
Oct 30 01:08:09.671501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.671448    5104 scope.go:122] "RemoveContainer" containerID="93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe"
Oct 30 01:08:09.673375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.673344    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:09.673513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.673433    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:09.673513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.673492    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:09.821132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.821081    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"]
Oct 30 01:08:09.822531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.822504    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fb1b67db-8039-4846-9897-f2f46846171e" containerName="revoke-keys"
Oct 30 01:08:09.822531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.822532    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1b67db-8039-4846-9897-f2f46846171e" containerName="revoke-keys"
Oct 30 01:08:09.822778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.822762    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1b67db-8039-4846-9897-f2f46846171e" containerName="revoke-keys"
Oct 30 01:08:09.852870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.852840    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:09.862561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.862384    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"]
Oct 30 01:08:09.872285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.872261    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-da60de\""
Oct 30 01:08:09.963887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.960683    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e21c9-07e5-44e8-9daf-08b012575a61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da60de-877f4976-zv2gh\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") " pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:09.964396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:09.964359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gklnl\" (UniqueName: \"kubernetes.io/projected/715e21c9-07e5-44e8-9daf-08b012575a61-kube-api-access-gklnl\") pod \"payload-processing-e2e-api-b-da60de-877f4976-zv2gh\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") " pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:10.082537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:10.079828    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklnl\" (UniqueName: \"kubernetes.io/projected/715e21c9-07e5-44e8-9daf-08b012575a61-kube-api-access-gklnl\") pod \"payload-processing-e2e-api-b-da60de-877f4976-zv2gh\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") " pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:10.082537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:10.080156    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e21c9-07e5-44e8-9daf-08b012575a61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da60de-877f4976-zv2gh\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") " pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:10.082537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:10.081213    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e21c9-07e5-44e8-9daf-08b012575a61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da60de-877f4976-zv2gh\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") " pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:10.171481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:10.160387    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gklnl\" (UniqueName: \"kubernetes.io/projected/715e21c9-07e5-44e8-9daf-08b012575a61-kube-api-access-gklnl\") pod \"payload-processing-e2e-api-b-da60de-877f4976-zv2gh\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") " pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:10.208816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:10.208772    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:10.934122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:10.932701    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" event={"ID":"fc10b078-6065-4c3b-9655-306808b43f55","Type":"ContainerStarted","Data":"011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a"}
Oct 30 01:08:10.986493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:10.983406    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"]
Oct 30 01:08:11.065290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.064744    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.096630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.081666    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:08:11.096630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.081790    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:08:11.096630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.081829    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"]
Oct 30 01:08:11.096630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.096426    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"]
Oct 30 01:08:11.129368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.129262    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:11.129368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.129319    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:08:11.236485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.231302    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.236485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.231517    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.236485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.231551    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grq7r\" (UniqueName: \"kubernetes.io/projected/a3a0f230-08dc-496e-b508-be4c7e908d42-kube-api-access-grq7r\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.345565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.340633    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.345565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.340819    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.345565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.340855    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grq7r\" (UniqueName: \"kubernetes.io/projected/a3a0f230-08dc-496e-b508-be4c7e908d42-kube-api-access-grq7r\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.345565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.344775    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.362475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.358863    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.371053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.370998    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grq7r\" (UniqueName: \"kubernetes.io/projected/a3a0f230-08dc-496e-b508-be4c7e908d42-kube-api-access-grq7r\") pod \"maas-api-revoke-keys-e2e-api-b-288da6-tzl2v\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.430483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.428771    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:08:11.432612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.432576    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:11.432861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.432835    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:11.433469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.433433    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:11.860716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.860676    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"]
Oct 30 01:08:11.864870 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:08:11.864806    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a0f230_08dc_496e_b508_be4c7e908d42.slice/crio-d88c37ead2d4b35e311428486a2bf14a1a7f26f79f3c903816bd82ec0a1280dc WatchSource:0}: Error finding container d88c37ead2d4b35e311428486a2bf14a1a7f26f79f3c903816bd82ec0a1280dc: Status 404 returned error can't find the container with id d88c37ead2d4b35e311428486a2bf14a1a7f26f79f3c903816bd82ec0a1280dc
Oct 30 01:08:11.943045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.943003    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh" event={"ID":"715e21c9-07e5-44e8-9daf-08b012575a61","Type":"ContainerStarted","Data":"7eb10c87ad16fd2166960830726ae09c6150e64442a23f9ef1e8d415c89cc979"}
Oct 30 01:08:11.943214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.943076    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh" event={"ID":"715e21c9-07e5-44e8-9daf-08b012575a61","Type":"ContainerStarted","Data":"68f83acdb344e3ee567f5c4e92ccb4775e675beec40245d71e628d3b44d20f9f"}
Oct 30 01:08:11.943214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.943108    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:11.944813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:11.944787    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v" event={"ID":"a3a0f230-08dc-496e-b508-be4c7e908d42","Type":"ContainerStarted","Data":"d88c37ead2d4b35e311428486a2bf14a1a7f26f79f3c903816bd82ec0a1280dc"}
Oct 30 01:08:12.011817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:12.011749    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh" podStartSLOduration=3.011730535 podStartE2EDuration="3.011730535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:11.983704463 +0000 UTC m=+1533.531394957" watchObservedRunningTime="2026-10-30 01:08:12.011730535 +0000 UTC m=+1533.559421040"
Oct 30 01:08:12.425471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:12.425427    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:12.425930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:12.425521    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:12.955604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:12.955565    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v" event={"ID":"a3a0f230-08dc-496e-b508-be4c7e908d42","Type":"ContainerStarted","Data":"3c4a719377817066357c7ef1c1ff08ab2661dd2b54f1d967b703f38869760bce"}
Oct 30 01:08:13.141647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:13.141588    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v" podStartSLOduration=3.14157261 podStartE2EDuration="3.14157261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:13.003836557 +0000 UTC m=+1534.551527053" watchObservedRunningTime="2026-10-30 01:08:13.14157261 +0000 UTC m=+1534.689263106"
Oct 30 01:08:13.146135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:13.146105    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"]
Oct 30 01:08:13.214760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:13.214725    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"]
Oct 30 01:08:13.459020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:13.432826    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:13.459020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:13.432891    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:14.079531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:14.061130    5104 generic.go:356] "Generic (PLEG): container finished" podID="a3a0f230-08dc-496e-b508-be4c7e908d42" containerID="3c4a719377817066357c7ef1c1ff08ab2661dd2b54f1d967b703f38869760bce" exitCode=0
Oct 30 01:08:14.079531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:14.061603    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v" event={"ID":"a3a0f230-08dc-496e-b508-be4c7e908d42","Type":"ContainerDied","Data":"3c4a719377817066357c7ef1c1ff08ab2661dd2b54f1d967b703f38869760bce"}
Oct 30 01:08:14.448493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:14.441717    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:14.448493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:14.441776    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:14.553410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:14.520584    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:14.553410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:14.520984    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:08:14.576915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:14.576864    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:15.436536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:15.426795    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:15.436536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:15.426852    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:16.372858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.372774    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:16.470479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.469969    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:16.470479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.470035    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:16.487477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.475053    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-xks-ca\") pod \"a3a0f230-08dc-496e-b508-be4c7e908d42\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") "
Oct 30 01:08:16.487477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.475174    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-openshift-ca\") pod \"a3a0f230-08dc-496e-b508-be4c7e908d42\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") "
Oct 30 01:08:16.487477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.475247    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a0f230-08dc-496e-b508-be4c7e908d42-kube-api-access-grq7r\" (UniqueName: \"kubernetes.io/projected/a3a0f230-08dc-496e-b508-be4c7e908d42-kube-api-access-grq7r\") pod \"a3a0f230-08dc-496e-b508-be4c7e908d42\" (UID: \"a3a0f230-08dc-496e-b508-be4c7e908d42\") "
Oct 30 01:08:16.490417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.489660    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-openshift-ca" pod "a3a0f230-08dc-496e-b508-be4c7e908d42" (UID: "a3a0f230-08dc-496e-b508-be4c7e908d42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:16.492204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.492171    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:16.530942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.530875    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a0f230-08dc-496e-b508-be4c7e908d42-kube-api-access-grq7r" pod "a3a0f230-08dc-496e-b508-be4c7e908d42" (UID: "a3a0f230-08dc-496e-b508-be4c7e908d42"). InnerVolumeSpecName "kube-api-access-grq7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:16.539994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.539917    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-xks-ca" pod "a3a0f230-08dc-496e-b508-be4c7e908d42" (UID: "a3a0f230-08dc-496e-b508-be4c7e908d42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:16.599182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.599142    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a0f230-08dc-496e-b508-be4c7e908d42-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:16.599496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:16.599442    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grq7r\" (UniqueName: \"kubernetes.io/projected/a3a0f230-08dc-496e-b508-be4c7e908d42-kube-api-access-grq7r\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:17.171437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:17.171407    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v" event={"ID":"a3a0f230-08dc-496e-b508-be4c7e908d42","Type":"ContainerDied","Data":"d88c37ead2d4b35e311428486a2bf14a1a7f26f79f3c903816bd82ec0a1280dc"}
Oct 30 01:08:17.171694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:17.171470    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d88c37ead2d4b35e311428486a2bf14a1a7f26f79f3c903816bd82ec0a1280dc"
Oct 30 01:08:17.171694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:17.171527    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"
Oct 30 01:08:17.435535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:17.435019    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:17.435535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:17.435089    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:18.156584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:18.156532    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f" podUID="0083837f-171e-42ef-a693-15f8952f21b7" containerName="payload-processing" containerID="cri-o://3d1d9c849f694106fc01143a124bc2b201e3559ebb6520d561cbd27548ecb312" gracePeriod=25
Oct 30 01:08:18.258531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:18.254796    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:08:18.258531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:18.255830    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z" podUID="2af0b59c-9cfa-4255-b94c-6473ae258b48" containerName="payload-pre-processing" containerID="cri-o://1dd9531e5ce18415f345f7e0f0e9d5619e2ca536e85efb75833dd55658d6b767" gracePeriod=25
Oct 30 01:08:18.428868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:18.428814    5104 patch_prober.go:29] interesting pod/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 01:08:18.429199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:18.429166    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 01:08:19.060647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:19.060590    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:19.079604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:19.079542    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:19.507714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:19.506502    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:08:19.507714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:19.506903    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:08:19.518001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:19.515600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:19.811553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:19.809089    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:20.066291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.064510    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:08:20.225207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.225163    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bq26k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:08:20.225530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.225494    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:20.353038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.352992    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:08:20.380482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.358212    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:08:20.429482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.413314    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:08:20.430693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.430661    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:08:20.464489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.462425    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:08:20.716482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.527044    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:08:20.716482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.642694    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:20.716482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.669167    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 30 01:08:20.716482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.672710    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:20.765250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.765219    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:08:20.815221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.815195    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:08:20.827391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.814114    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:08:20.828391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.827670    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:20.852892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.827777    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:08:20.853139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.853124    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:08:20.955488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:20.954942    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:08:21.065470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.065367    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af0b59c-9cfa-4255-b94c-6473ae258b48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af0b59c-9cfa-4255-b94c-6473ae258b48-plugins-config-volume\") pod \"2af0b59c-9cfa-4255-b94c-6473ae258b48\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") "
Oct 30 01:08:21.069490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.066636    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0083837f-171e-42ef-a693-15f8952f21b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0083837f-171e-42ef-a693-15f8952f21b7-plugins-config-volume\") pod \"0083837f-171e-42ef-a693-15f8952f21b7\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") "
Oct 30 01:08:21.092711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.081240    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af0b59c-9cfa-4255-b94c-6473ae258b48-plugins-config-volume" pod "2af0b59c-9cfa-4255-b94c-6473ae258b48" (UID: "2af0b59c-9cfa-4255-b94c-6473ae258b48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:21.092951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.081500    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0083837f-171e-42ef-a693-15f8952f21b7-plugins-config-volume" pod "0083837f-171e-42ef-a693-15f8952f21b7" (UID: "0083837f-171e-42ef-a693-15f8952f21b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:21.130493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.117693    5104 generic.go:356] "Generic (PLEG): container finished" podID="0083837f-171e-42ef-a693-15f8952f21b7" containerID="3d1d9c849f694106fc01143a124bc2b201e3559ebb6520d561cbd27548ecb312" exitCode=0
Oct 30 01:08:21.140006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.139969    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0083837f-171e-42ef-a693-15f8952f21b7-kube-api-access-8twmr\" (UniqueName: \"kubernetes.io/projected/0083837f-171e-42ef-a693-15f8952f21b7-kube-api-access-8twmr\") pod \"0083837f-171e-42ef-a693-15f8952f21b7\" (UID: \"0083837f-171e-42ef-a693-15f8952f21b7\") "
Oct 30 01:08:21.140610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.140578    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af0b59c-9cfa-4255-b94c-6473ae258b48-kube-api-access-dw9qj\" (UniqueName: \"kubernetes.io/projected/2af0b59c-9cfa-4255-b94c-6473ae258b48-kube-api-access-dw9qj\") pod \"2af0b59c-9cfa-4255-b94c-6473ae258b48\" (UID: \"2af0b59c-9cfa-4255-b94c-6473ae258b48\") "
Oct 30 01:08:21.189263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.189232    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af0b59c-9cfa-4255-b94c-6473ae258b48-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:21.192695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.192672    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0083837f-171e-42ef-a693-15f8952f21b7-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:21.206598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.206566    5104 generic.go:356] "Generic (PLEG): container finished" podID="2af0b59c-9cfa-4255-b94c-6473ae258b48" containerID="1dd9531e5ce18415f345f7e0f0e9d5619e2ca536e85efb75833dd55658d6b767" exitCode=0
Oct 30 01:08:21.208700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.208661    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"
Oct 30 01:08:21.247579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.247554    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:08:21.247862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.247824    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f" event={"ID":"0083837f-171e-42ef-a693-15f8952f21b7","Type":"ContainerDied","Data":"3d1d9c849f694106fc01143a124bc2b201e3559ebb6520d561cbd27548ecb312"}
Oct 30 01:08:21.248117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.248102    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:08:21.248324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.248247    5104 scope.go:122] "RemoveContainer" containerID="3d1d9c849f694106fc01143a124bc2b201e3559ebb6520d561cbd27548ecb312"
Oct 30 01:08:21.250347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.248295    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z" event={"ID":"2af0b59c-9cfa-4255-b94c-6473ae258b48","Type":"ContainerDied","Data":"1dd9531e5ce18415f345f7e0f0e9d5619e2ca536e85efb75833dd55658d6b767"}
Oct 30 01:08:21.250347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.250206    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:08:21.250347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.250245    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:08:21.250347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.250290    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:08:21.354121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.349866    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0083837f-171e-42ef-a693-15f8952f21b7-kube-api-access-8twmr" pod "0083837f-171e-42ef-a693-15f8952f21b7" (UID: "0083837f-171e-42ef-a693-15f8952f21b7"). InnerVolumeSpecName "kube-api-access-8twmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:21.464544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.461150    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af0b59c-9cfa-4255-b94c-6473ae258b48-kube-api-access-dw9qj" pod "2af0b59c-9cfa-4255-b94c-6473ae258b48" (UID: "2af0b59c-9cfa-4255-b94c-6473ae258b48"). InnerVolumeSpecName "kube-api-access-dw9qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:21.472781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.468806    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twmr\" (UniqueName: \"kubernetes.io/projected/0083837f-171e-42ef-a693-15f8952f21b7-kube-api-access-8twmr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:21.472781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.468844    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9qj\" (UniqueName: \"kubernetes.io/projected/2af0b59c-9cfa-4255-b94c-6473ae258b48-kube-api-access-dw9qj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:21.649479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.647891    5104 scope.go:122] "RemoveContainer" containerID="1dd9531e5ce18415f345f7e0f0e9d5619e2ca536e85efb75833dd55658d6b767"
Oct 30 01:08:21.813500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.813144    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"]
Oct 30 01:08:21.821814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:21.821709    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-260268-89664cb59-4p45z"]
Oct 30 01:08:22.218833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.218804    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"
Oct 30 01:08:22.219334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.218905    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f" event={"ID":"0083837f-171e-42ef-a693-15f8952f21b7","Type":"ContainerDied","Data":"4f64e19dba3d3e2fdd46318cd0cea7072995b961b30a1bce64b6c44b14e1e607"}
Oct 30 01:08:22.270893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.270851    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"]
Oct 30 01:08:22.275872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.275848    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-260268-5558bb6674-bwd6f"]
Oct 30 01:08:22.554601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.554517    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"]
Oct 30 01:08:22.585229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.585197    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0083837f-171e-42ef-a693-15f8952f21b7" path="/var/lib/kubelet/pods/0083837f-171e-42ef-a693-15f8952f21b7/volumes"
Oct 30 01:08:22.585867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.585844    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af0b59c-9cfa-4255-b94c-6473ae258b48" path="/var/lib/kubelet/pods/2af0b59c-9cfa-4255-b94c-6473ae258b48/volumes"
Oct 30 01:08:22.959325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:22.959296    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:08:27.555555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.555490    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79" podUID="093e06f8-d249-4474-857d-4818e3df198d" containerName="payload-pre-processing" containerID="cri-o://69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f" gracePeriod=25
Oct 30 01:08:27.623988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.623951    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"]
Oct 30 01:08:27.624866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.624849    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a3a0f230-08dc-496e-b508-be4c7e908d42" containerName="revoke-keys"
Oct 30 01:08:27.624935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.624868    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a0f230-08dc-496e-b508-be4c7e908d42" containerName="revoke-keys"
Oct 30 01:08:27.624935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.624890    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2af0b59c-9cfa-4255-b94c-6473ae258b48" containerName="payload-pre-processing"
Oct 30 01:08:27.624935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.624895    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af0b59c-9cfa-4255-b94c-6473ae258b48" containerName="payload-pre-processing"
Oct 30 01:08:27.624935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.624913    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0083837f-171e-42ef-a693-15f8952f21b7" containerName="payload-processing"
Oct 30 01:08:27.624935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.624919    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0083837f-171e-42ef-a693-15f8952f21b7" containerName="payload-processing"
Oct 30 01:08:27.625096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.625078    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af0b59c-9cfa-4255-b94c-6473ae258b48" containerName="payload-pre-processing"
Oct 30 01:08:27.625096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.625089    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0083837f-171e-42ef-a693-15f8952f21b7" containerName="payload-processing"
Oct 30 01:08:27.625158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.625104    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a0f230-08dc-496e-b508-be4c7e908d42" containerName="revoke-keys"
Oct 30 01:08:27.629280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.629259    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.646555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.646519    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"]
Oct 30 01:08:27.652568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.652532    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:08:27.654206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.654186    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:08:27.654206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.654192    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:27.654376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.654205    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:08:27.739238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.739199    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.739489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.739451    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvhf\" (UniqueName: \"kubernetes.io/projected/2432fa9c-ff6f-4ffb-a51d-45537b200a64-kube-api-access-ldvhf\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.739658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.739642    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.841061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.840967    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvhf\" (UniqueName: \"kubernetes.io/projected/2432fa9c-ff6f-4ffb-a51d-45537b200a64-kube-api-access-ldvhf\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.841225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.841200    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.841385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.841331    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.842708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.842670    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.846922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.846876    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.864183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.864143    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvhf\" (UniqueName: \"kubernetes.io/projected/2432fa9c-ff6f-4ffb-a51d-45537b200a64-kube-api-access-ldvhf\") pod \"maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:27.909408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.909379    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:08:27.943749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.943719    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093e06f8-d249-4474-857d-4818e3df198d-kube-api-access-2bmhf\" (UniqueName: \"kubernetes.io/projected/093e06f8-d249-4474-857d-4818e3df198d-kube-api-access-2bmhf\") pod \"093e06f8-d249-4474-857d-4818e3df198d\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") "
Oct 30 01:08:27.943934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.943782    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093e06f8-d249-4474-857d-4818e3df198d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093e06f8-d249-4474-857d-4818e3df198d-plugins-config-volume\") pod \"093e06f8-d249-4474-857d-4818e3df198d\" (UID: \"093e06f8-d249-4474-857d-4818e3df198d\") "
Oct 30 01:08:27.944330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.944236    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093e06f8-d249-4474-857d-4818e3df198d-plugins-config-volume" pod "093e06f8-d249-4474-857d-4818e3df198d" (UID: "093e06f8-d249-4474-857d-4818e3df198d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:27.944712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.944687    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093e06f8-d249-4474-857d-4818e3df198d-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:27.946351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.946311    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093e06f8-d249-4474-857d-4818e3df198d-kube-api-access-2bmhf" pod "093e06f8-d249-4474-857d-4818e3df198d" (UID: "093e06f8-d249-4474-857d-4818e3df198d"). InnerVolumeSpecName "kube-api-access-2bmhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:27.974433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:27.974366    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:28.046702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.046656    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmhf\" (UniqueName: \"kubernetes.io/projected/093e06f8-d249-4474-857d-4818e3df198d-kube-api-access-2bmhf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:28.299634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.299277    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"]
Oct 30 01:08:28.308970 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:08:28.308925    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2432fa9c_ff6f_4ffb_a51d_45537b200a64.slice/crio-7739c5d937676dc659418794a638cad8e7f088933f69beed08967cdf99ddb02b WatchSource:0}: Error finding container 7739c5d937676dc659418794a638cad8e7f088933f69beed08967cdf99ddb02b: Status 404 returned error can't find the container with id 7739c5d937676dc659418794a638cad8e7f088933f69beed08967cdf99ddb02b
Oct 30 01:08:28.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.430218    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv" event={"ID":"2432fa9c-ff6f-4ffb-a51d-45537b200a64","Type":"ContainerStarted","Data":"7739c5d937676dc659418794a638cad8e7f088933f69beed08967cdf99ddb02b"}
Oct 30 01:08:28.431844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.431819    5104 generic.go:356] "Generic (PLEG): container finished" podID="093e06f8-d249-4474-857d-4818e3df198d" containerID="69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f" exitCode=0
Oct 30 01:08:28.431947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.431885    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"
Oct 30 01:08:28.431947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.431901    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79" event={"ID":"093e06f8-d249-4474-857d-4818e3df198d","Type":"ContainerDied","Data":"69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f"}
Oct 30 01:08:28.432044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.431963    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79" event={"ID":"093e06f8-d249-4474-857d-4818e3df198d","Type":"ContainerDied","Data":"d232b137c6b66dbfa1b7e49e7d7a95d4fe16d217c98e027f553b2f510c1542ad"}
Oct 30 01:08:28.432044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.431994    5104 scope.go:122] "RemoveContainer" containerID="69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f"
Oct 30 01:08:28.445369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.445286    5104 scope.go:122] "RemoveContainer" containerID="69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f"
Oct 30 01:08:28.445724 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:08:28.445691    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f\": container with ID starting with 69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f not found: ID does not exist" containerID="69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f"
Oct 30 01:08:28.445852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.445735    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f"} err="failed to get container status \"69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f\": rpc error: code = NotFound desc = could not find container \"69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f\": container with ID starting with 69f0ddd9e4f0fb58084bcc63db720df4493a32586145c9a45481e76b0240030f not found: ID does not exist"
Oct 30 01:08:28.485333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.485301    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"]
Oct 30 01:08:28.500220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.500185    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-288da6-7d5db4cc67-9xs79"]
Oct 30 01:08:28.581842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:28.581763    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093e06f8-d249-4474-857d-4818e3df198d" path="/var/lib/kubelet/pods/093e06f8-d249-4474-857d-4818e3df198d/volumes"
Oct 30 01:08:29.522181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:29.522125    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv" event={"ID":"2432fa9c-ff6f-4ffb-a51d-45537b200a64","Type":"ContainerStarted","Data":"6fa1c10c73c5f949bbff9676968c9e17abdf0e7e9333a662a99bd2495ad66779"}
Oct 30 01:08:29.561195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:29.561143    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv" podStartSLOduration=2.561126617 podStartE2EDuration="2.561126617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:29.560692386 +0000 UTC m=+1551.108382884" watchObservedRunningTime="2026-10-30 01:08:29.561126617 +0000 UTC m=+1551.108817114"
Oct 30 01:08:30.581476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:30.565261    5104 generic.go:356] "Generic (PLEG): container finished" podID="2432fa9c-ff6f-4ffb-a51d-45537b200a64" containerID="6fa1c10c73c5f949bbff9676968c9e17abdf0e7e9333a662a99bd2495ad66779" exitCode=0
Oct 30 01:08:30.586303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:30.575215    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv" event={"ID":"2432fa9c-ff6f-4ffb-a51d-45537b200a64","Type":"ContainerDied","Data":"6fa1c10c73c5f949bbff9676968c9e17abdf0e7e9333a662a99bd2495ad66779"}
Oct 30 01:08:31.957920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:31.957889    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:32.075947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.075914    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-xks-ca\") pod \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") "
Oct 30 01:08:32.076261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.076238    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-openshift-ca\") pod \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") "
Oct 30 01:08:32.076364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.076303    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2432fa9c-ff6f-4ffb-a51d-45537b200a64-kube-api-access-ldvhf\" (UniqueName: \"kubernetes.io/projected/2432fa9c-ff6f-4ffb-a51d-45537b200a64-kube-api-access-ldvhf\") pod \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\" (UID: \"2432fa9c-ff6f-4ffb-a51d-45537b200a64\") "
Oct 30 01:08:32.076694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.076667    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-openshift-ca" pod "2432fa9c-ff6f-4ffb-a51d-45537b200a64" (UID: "2432fa9c-ff6f-4ffb-a51d-45537b200a64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:32.077059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.077034    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:32.079323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.079297    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2432fa9c-ff6f-4ffb-a51d-45537b200a64-kube-api-access-ldvhf" pod "2432fa9c-ff6f-4ffb-a51d-45537b200a64" (UID: "2432fa9c-ff6f-4ffb-a51d-45537b200a64"). InnerVolumeSpecName "kube-api-access-ldvhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:32.090692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.090657    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-xks-ca" pod "2432fa9c-ff6f-4ffb-a51d-45537b200a64" (UID: "2432fa9c-ff6f-4ffb-a51d-45537b200a64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:32.180502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.179486    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvhf\" (UniqueName: \"kubernetes.io/projected/2432fa9c-ff6f-4ffb-a51d-45537b200a64-kube-api-access-ldvhf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:32.180502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.179521    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2432fa9c-ff6f-4ffb-a51d-45537b200a64-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:32.626980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.626944    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"
Oct 30 01:08:32.627130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.626976    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv" event={"ID":"2432fa9c-ff6f-4ffb-a51d-45537b200a64","Type":"ContainerDied","Data":"7739c5d937676dc659418794a638cad8e7f088933f69beed08967cdf99ddb02b"}
Oct 30 01:08:32.627130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:32.627026    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7739c5d937676dc659418794a638cad8e7f088933f69beed08967cdf99ddb02b"
Oct 30 01:08:36.018479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:36.018136    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:08:38.337388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:38.337351    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"]
Oct 30 01:08:38.377050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:38.377002    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"]
Oct 30 01:08:39.133008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:39.113534    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"]
Oct 30 01:08:39.157269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:39.136959    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9711d-cfhkx"]
Oct 30 01:08:39.378483 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:08:39.378015    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:08:40.633998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:40.633059    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e6acec-5c73-4257-a7bb-90ebf9a2dd52" path="/var/lib/kubelet/pods/61e6acec-5c73-4257-a7bb-90ebf9a2dd52/volumes"
Oct 30 01:08:43.028170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.028127    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"]
Oct 30 01:08:43.038377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.038344    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbce7c5-gxk7s"]
Oct 30 01:08:43.352836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.352725    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h" podUID="02bdbe07-8a8b-4ef4-81af-02f4e1423963" containerName="payload-processing" containerID="cri-o://1185fca50950dda07e119af4bb5bc008922292ed8632a1ff8514b50470d6e2d3" gracePeriod=25
Oct 30 01:08:43.355050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.355023    5104 scope.go:122] "RemoveContainer" containerID="96ac9371c1d399899035077fedcef07d70523766fb68c160391b4b0721b45cb0"
Oct 30 01:08:43.378065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.373661    5104 scope.go:122] "RemoveContainer" containerID="006e2382e359f0ba5aa7d9f82b8c34e787ec2dc983106f197174f272cbea0ba1"
Oct 30 01:08:43.378065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.377856    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr" podUID="5c20b9ac-bd1a-40de-a621-55c5ef126d96" containerName="payload-pre-processing" containerID="cri-o://48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07" gracePeriod=25
Oct 30 01:08:43.514685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.514247    5104 scope.go:122] "RemoveContainer" containerID="bd25f3a3684aec8cc28b9a427b7e85f030ed9a9f80f98ba78312cd9ca3ad3f15"
Oct 30 01:08:43.529765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.529743    5104 scope.go:122] "RemoveContainer" containerID="a3abe2b1138a443ed4e82e809172940e66b628319fc30a20e088908ed2ac71f7"
Oct 30 01:08:43.544012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.543985    5104 scope.go:122] "RemoveContainer" containerID="a753de4e01f2296c14990c34fedeb392e817dc1838d9212f7f062aa4a6a2f311"
Oct 30 01:08:43.831979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.827841    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:08:43.922531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.912586    5104 generic.go:356] "Generic (PLEG): container finished" podID="02bdbe07-8a8b-4ef4-81af-02f4e1423963" containerID="1185fca50950dda07e119af4bb5bc008922292ed8632a1ff8514b50470d6e2d3" exitCode=0
Oct 30 01:08:43.922531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.912919    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h" event={"ID":"02bdbe07-8a8b-4ef4-81af-02f4e1423963","Type":"ContainerDied","Data":"1185fca50950dda07e119af4bb5bc008922292ed8632a1ff8514b50470d6e2d3"}
Oct 30 01:08:43.922531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.916822    5104 generic.go:356] "Generic (PLEG): container finished" podID="5c20b9ac-bd1a-40de-a621-55c5ef126d96" containerID="48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07" exitCode=0
Oct 30 01:08:43.922531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.917064    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr" event={"ID":"5c20b9ac-bd1a-40de-a621-55c5ef126d96","Type":"ContainerDied","Data":"48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07"}
Oct 30 01:08:43.922531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.917116    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr" event={"ID":"5c20b9ac-bd1a-40de-a621-55c5ef126d96","Type":"ContainerDied","Data":"a6c89e36d4189bad449274e43638ae165ae94988d6775920ed0acd7263d92440"}
Oct 30 01:08:43.922531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.917163    5104 scope.go:122] "RemoveContainer" containerID="48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07"
Oct 30 01:08:43.922531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.917336    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"
Oct 30 01:08:43.951150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.950267    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c20b9ac-bd1a-40de-a621-55c5ef126d96-kube-api-access-2cd6x\" (UniqueName: \"kubernetes.io/projected/5c20b9ac-bd1a-40de-a621-55c5ef126d96-kube-api-access-2cd6x\") pod \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") "
Oct 30 01:08:43.951150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.950438    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c20b9ac-bd1a-40de-a621-55c5ef126d96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c20b9ac-bd1a-40de-a621-55c5ef126d96-plugins-config-volume\") pod \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\" (UID: \"5c20b9ac-bd1a-40de-a621-55c5ef126d96\") "
Oct 30 01:08:43.952529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.952392    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c20b9ac-bd1a-40de-a621-55c5ef126d96-plugins-config-volume" pod "5c20b9ac-bd1a-40de-a621-55c5ef126d96" (UID: "5c20b9ac-bd1a-40de-a621-55c5ef126d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:43.958031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:43.957987    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c20b9ac-bd1a-40de-a621-55c5ef126d96-kube-api-access-2cd6x" pod "5c20b9ac-bd1a-40de-a621-55c5ef126d96" (UID: "5c20b9ac-bd1a-40de-a621-55c5ef126d96"). InnerVolumeSpecName "kube-api-access-2cd6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:44.030801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.030696    5104 scope.go:122] "RemoveContainer" containerID="48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07"
Oct 30 01:08:44.031828 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:08:44.031800    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07\": container with ID starting with 48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07 not found: ID does not exist" containerID="48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07"
Oct 30 01:08:44.032008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.031978    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07"} err="failed to get container status \"48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07\": rpc error: code = NotFound desc = could not find container \"48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07\": container with ID starting with 48675a151ed6034c99075feed007b03b764124fa3aeaf2468d7954347bbfdb07 not found: ID does not exist"
Oct 30 01:08:44.036773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.036479    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:08:44.053566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.053529    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cd6x\" (UniqueName: \"kubernetes.io/projected/5c20b9ac-bd1a-40de-a621-55c5ef126d96-kube-api-access-2cd6x\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:44.053566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.053565    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c20b9ac-bd1a-40de-a621-55c5ef126d96-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:44.178482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.170944    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bdbe07-8a8b-4ef4-81af-02f4e1423963-kube-api-access-nkdpv\" (UniqueName: \"kubernetes.io/projected/02bdbe07-8a8b-4ef4-81af-02f4e1423963-kube-api-access-nkdpv\") pod \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") "
Oct 30 01:08:44.178482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.171321    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bdbe07-8a8b-4ef4-81af-02f4e1423963-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bdbe07-8a8b-4ef4-81af-02f4e1423963-plugins-config-volume\") pod \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\" (UID: \"02bdbe07-8a8b-4ef4-81af-02f4e1423963\") "
Oct 30 01:08:44.260104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.260043    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bdbe07-8a8b-4ef4-81af-02f4e1423963-plugins-config-volume" pod "02bdbe07-8a8b-4ef4-81af-02f4e1423963" (UID: "02bdbe07-8a8b-4ef4-81af-02f4e1423963"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:44.465123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.390554    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bdbe07-8a8b-4ef4-81af-02f4e1423963-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:44.465123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.390604    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"]
Oct 30 01:08:44.470419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.470385    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb3370cb-5569c6b947-j9fwr"]
Oct 30 01:08:44.471342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.471294    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bdbe07-8a8b-4ef4-81af-02f4e1423963-kube-api-access-nkdpv" pod "02bdbe07-8a8b-4ef4-81af-02f4e1423963" (UID: "02bdbe07-8a8b-4ef4-81af-02f4e1423963"). InnerVolumeSpecName "kube-api-access-nkdpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:44.510778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.510643    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkdpv\" (UniqueName: \"kubernetes.io/projected/02bdbe07-8a8b-4ef4-81af-02f4e1423963-kube-api-access-nkdpv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:08:44.613214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.613179    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c20b9ac-bd1a-40de-a621-55c5ef126d96" path="/var/lib/kubelet/pods/5c20b9ac-bd1a-40de-a621-55c5ef126d96/volumes"
Oct 30 01:08:44.613702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:44.613686    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772ec5e3-3c34-4742-8384-bcdee2872a20" path="/var/lib/kubelet/pods/772ec5e3-3c34-4742-8384-bcdee2872a20/volumes"
Oct 30 01:08:45.159289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:45.159246    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"
Oct 30 01:08:45.163510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:45.160700    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h" event={"ID":"02bdbe07-8a8b-4ef4-81af-02f4e1423963","Type":"ContainerDied","Data":"0ad189c97aaf17c9f53bdd64bfa2cf5d3162ded07b3c51023f6da845072dc5b3"}
Oct 30 01:08:45.163510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:45.160816    5104 scope.go:122] "RemoveContainer" containerID="1185fca50950dda07e119af4bb5bc008922292ed8632a1ff8514b50470d6e2d3"
Oct 30 01:08:45.230629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:45.230586    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"]
Oct 30 01:08:45.264483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:45.256260    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-2686h"]
Oct 30 01:08:46.584653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:46.584561    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bdbe07-8a8b-4ef4-81af-02f4e1423963" path="/var/lib/kubelet/pods/02bdbe07-8a8b-4ef4-81af-02f4e1423963/volumes"
Oct 30 01:08:49.027929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:49.027824    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"]
Oct 30 01:08:49.038424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:49.038391    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5709a163-f72v4"]
Oct 30 01:08:50.584634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:50.582563    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7456860a-d5c7-4b4d-ae07-f8ff42129f7b" path="/var/lib/kubelet/pods/7456860a-d5c7-4b4d-ae07-f8ff42129f7b/volumes"
Oct 30 01:08:51.237700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:51.237669    5104 generic.go:356] "Generic (PLEG): container finished" podID="1388f407-8532-47c6-b310-57e2c3302a07" containerID="d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91" exitCode=0
Oct 30 01:08:51.237893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:51.237742    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" event={"ID":"1388f407-8532-47c6-b310-57e2c3302a07","Type":"ContainerDied","Data":"d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91"}
Oct 30 01:08:51.238275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:51.238261    5104 scope.go:122] "RemoveContainer" containerID="d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91"
Oct 30 01:08:51.240192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:51.240163    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:51.240266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:51.240240    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:51.240303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:51.240272    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:08:52.249277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:52.249244    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" event={"ID":"1388f407-8532-47c6-b310-57e2c3302a07","Type":"ContainerStarted","Data":"8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e"}
Oct 30 01:08:52.559214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:52.559119    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:08:52.560329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:52.560299    5104 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:52.560477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:52.560378    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:53.560022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:53.559988    5104 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:53.560429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:53.560040    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:54.559972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:54.559864    5104 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:54.560283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:54.560253    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:55.110491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.107560    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"]
Oct 30 01:08:55.385539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.349040    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:08:55.429003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.428965    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb3370cb-nnvrv"]
Oct 30 01:08:55.576190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.576129    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:55.577562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.577533    5104 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:55.620289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.620236    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:55.641668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.641576    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"]
Oct 30 01:08:55.641996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.641977    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-260268-nnfkt"]
Oct 30 01:08:55.661164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.660749    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:08:55.677087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.676497    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:55.734147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:55.734104    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:56.559537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:56.559503    5104 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:56.559723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:56.559558    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:56.613993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:56.613950    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2432fa9c-ff6f-4ffb-a51d-45537b200a64" path="/var/lib/kubelet/pods/2432fa9c-ff6f-4ffb-a51d-45537b200a64/volumes"
Oct 30 01:08:56.614576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:56.614557    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1b67db-8039-4846-9897-f2f46846171e" path="/var/lib/kubelet/pods/fb1b67db-8039-4846-9897-f2f46846171e/volumes"
Oct 30 01:08:57.559906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:57.559872    5104 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:57.560092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:57.559927    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:58.559769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:58.559729    5104 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:08:58.560206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:58.559799    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 01:08:59.564947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:59.564918    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:08:59.566288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:08:59.566265    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:09:00.264922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:00.264884    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"]
Oct 30 01:09:02.450187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.450146    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"]
Oct 30 01:09:02.450773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.450437    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" containerID="cri-o://5d16ccf4848fc7e79b0f97af660da0fe5adce933b14a9ec6d118a6fc92272d16" gracePeriod=30
Oct 30 01:09:02.965716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.965664    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"]
Oct 30 01:09:02.967491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.967417    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2432fa9c-ff6f-4ffb-a51d-45537b200a64" containerName="revoke-keys"
Oct 30 01:09:02.967766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.967746    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2432fa9c-ff6f-4ffb-a51d-45537b200a64" containerName="revoke-keys"
Oct 30 01:09:02.967906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.967895    5104 cpu_manager.go:414] "containerMap: removing container" podUID="093e06f8-d249-4474-857d-4818e3df198d" containerName="payload-pre-processing"
Oct 30 01:09:02.967987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.967976    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093e06f8-d249-4474-857d-4818e3df198d" containerName="payload-pre-processing"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.968797    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5c20b9ac-bd1a-40de-a621-55c5ef126d96" containerName="payload-pre-processing"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.968826    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c20b9ac-bd1a-40de-a621-55c5ef126d96" containerName="payload-pre-processing"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.968858    5104 cpu_manager.go:414] "containerMap: removing container" podUID="02bdbe07-8a8b-4ef4-81af-02f4e1423963" containerName="payload-processing"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.968864    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bdbe07-8a8b-4ef4-81af-02f4e1423963" containerName="payload-processing"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.969163    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c20b9ac-bd1a-40de-a621-55c5ef126d96" containerName="payload-pre-processing"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.969183    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bdbe07-8a8b-4ef4-81af-02f4e1423963" containerName="payload-processing"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.969198    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2432fa9c-ff6f-4ffb-a51d-45537b200a64" containerName="revoke-keys"
Oct 30 01:09:02.972526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.969212    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="093e06f8-d249-4474-857d-4818e3df198d" containerName="payload-pre-processing"
Oct 30 01:09:02.975164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:02.975136    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.006639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.002149    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:03.006639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.003088    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:03.006639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.003219    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:09:03.033554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.029546    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:03.033554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.032316    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"]
Oct 30 01:09:03.276581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.276485    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.276961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.276921    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.277237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.277209    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvh2s\" (UniqueName: \"kubernetes.io/projected/a6a12606-b93f-4352-927a-2cdf6b5a70f5-kube-api-access-kvh2s\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.405481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.404847    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.405695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.405493    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.405695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.405594    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvh2s\" (UniqueName: \"kubernetes.io/projected/a6a12606-b93f-4352-927a-2cdf6b5a70f5-kube-api-access-kvh2s\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.406543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.406521    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.408068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.408043    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.427766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.427737    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvh2s\" (UniqueName: \"kubernetes.io/projected/a6a12606-b93f-4352-927a-2cdf6b5a70f5-kube-api-access-kvh2s\") pod \"maas-api-revoke-keys-e2e-api-b-07410c-fcp9d\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.535520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.535393    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:03.905477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:03.895741    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"]
Oct 30 01:09:04.437419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:04.437382    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d" event={"ID":"a6a12606-b93f-4352-927a-2cdf6b5a70f5","Type":"ContainerStarted","Data":"3a09b8d09e6fe86c7eb0bbe04bafadd63b69923cc72f64326cb5aa45d8897327"}
Oct 30 01:09:04.437612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:04.437444    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d" event={"ID":"a6a12606-b93f-4352-927a-2cdf6b5a70f5","Type":"ContainerStarted","Data":"e7f6b0403c1da20fdb75862e2e2bb1d8094e9578cab35ca3f866c0ea3c5a6041"}
Oct 30 01:09:04.485481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:04.478525    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d" podStartSLOduration=2.478504478 podStartE2EDuration="2.478504478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:04.476913369 +0000 UTC m=+1586.024603867" watchObservedRunningTime="2026-10-30 01:09:04.478504478 +0000 UTC m=+1586.026194974"
Oct 30 01:09:05.448803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:05.448774    5104 generic.go:356] "Generic (PLEG): container finished" podID="a6a12606-b93f-4352-927a-2cdf6b5a70f5" containerID="3a09b8d09e6fe86c7eb0bbe04bafadd63b69923cc72f64326cb5aa45d8897327" exitCode=0
Oct 30 01:09:05.449233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:05.448860    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d" event={"ID":"a6a12606-b93f-4352-927a-2cdf6b5a70f5","Type":"ContainerDied","Data":"3a09b8d09e6fe86c7eb0bbe04bafadd63b69923cc72f64326cb5aa45d8897327"}
Oct 30 01:09:06.645985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:06.644127    5104 patch_prober.go:29] interesting pod/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:09:06.645985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:06.644212    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:09:07.134248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.134210    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:07.246832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.246799    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a12606-b93f-4352-927a-2cdf6b5a70f5-kube-api-access-kvh2s\" (UniqueName: \"kubernetes.io/projected/a6a12606-b93f-4352-927a-2cdf6b5a70f5-kube-api-access-kvh2s\") pod \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") "
Oct 30 01:09:07.247082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.246848    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-xks-ca\") pod \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") "
Oct 30 01:09:07.247082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.246946    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-openshift-ca\") pod \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\" (UID: \"a6a12606-b93f-4352-927a-2cdf6b5a70f5\") "
Oct 30 01:09:07.247994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.247963    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-openshift-ca" pod "a6a12606-b93f-4352-927a-2cdf6b5a70f5" (UID: "a6a12606-b93f-4352-927a-2cdf6b5a70f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:07.250241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.250216    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a12606-b93f-4352-927a-2cdf6b5a70f5-kube-api-access-kvh2s" pod "a6a12606-b93f-4352-927a-2cdf6b5a70f5" (UID: "a6a12606-b93f-4352-927a-2cdf6b5a70f5"). InnerVolumeSpecName "kube-api-access-kvh2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:07.251136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.251113    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-xks-ca" pod "a6a12606-b93f-4352-927a-2cdf6b5a70f5" (UID: "a6a12606-b93f-4352-927a-2cdf6b5a70f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:07.369535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.367017    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvh2s\" (UniqueName: \"kubernetes.io/projected/a6a12606-b93f-4352-927a-2cdf6b5a70f5-kube-api-access-kvh2s\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:07.369535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.367059    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:07.369535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.367080    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a12606-b93f-4352-927a-2cdf6b5a70f5-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:07.471066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.471029    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d" event={"ID":"a6a12606-b93f-4352-927a-2cdf6b5a70f5","Type":"ContainerDied","Data":"e7f6b0403c1da20fdb75862e2e2bb1d8094e9578cab35ca3f866c0ea3c5a6041"}
Oct 30 01:09:07.471245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.471094    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f6b0403c1da20fdb75862e2e2bb1d8094e9578cab35ca3f866c0ea3c5a6041"
Oct 30 01:09:07.471245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:07.471166    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"
Oct 30 01:09:08.644482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.640739    5104 generic.go:356] "Generic (PLEG): container finished" podID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerID="5d16ccf4848fc7e79b0f97af660da0fe5adce933b14a9ec6d118a6fc92272d16" exitCode=0
Oct 30 01:09:08.677591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.672446    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" event={"ID":"a76516bd-ba32-46c7-8a0f-878527bc1633","Type":"ContainerDied","Data":"5d16ccf4848fc7e79b0f97af660da0fe5adce933b14a9ec6d118a6fc92272d16"}
Oct 30 01:09:08.677591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.672534    5104 scope.go:122] "RemoveContainer" containerID="8b913aab1f84d28d56ead6e21e01913b007640830de5fb1aa93f591200d44ace"
Oct 30 01:09:08.740338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.740304    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:09:08.819728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.819674    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-podinfo\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.820096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.820067    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-socket\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.820447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.820416    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76516bd-ba32-46c7-8a0f-878527bc1633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76516bd-ba32-46c7-8a0f-878527bc1633-istiod-ca-cert\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.820752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.820733    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-data\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.820992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.820974    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-credential-socket\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.821272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.821240    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-kube-api-access-hsjqm\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-kube-api-access-hsjqm\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.821442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.821428    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-certs\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.821645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.821622    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-envoy\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.832838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.832804    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-certs" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:08.843028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.841892    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-data" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:08.843028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.842006    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-socket" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:08.843688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.843652    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-credential-socket" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:08.844392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.844362    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-token\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-token\") pod \"a76516bd-ba32-46c7-8a0f-878527bc1633\" (UID: \"a76516bd-ba32-46c7-8a0f-878527bc1633\") "
Oct 30 01:09:08.852329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.852299    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76516bd-ba32-46c7-8a0f-878527bc1633-istiod-ca-cert" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:08.890219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.890190    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:08.890400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.890386    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:08.907599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.907557    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76516bd-ba32-46c7-8a0f-878527bc1633-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:08.907880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.907864    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:08.908126 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.908107    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:08.912732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.912673    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-token" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:08.923804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.922704    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-kube-api-access-hsjqm" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "kube-api-access-hsjqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:08.936271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.934227    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-podinfo" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:08.936271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:08.934233    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-envoy" pod "a76516bd-ba32-46c7-8a0f-878527bc1633" (UID: "a76516bd-ba32-46c7-8a0f-878527bc1633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:09.010794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.010750    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsjqm\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-kube-api-access-hsjqm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:09.010794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.010796    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:09.011054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.010816    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:09.011054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.010829    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76516bd-ba32-46c7-8a0f-878527bc1633-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:09.253813 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:09:09.252797    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:09:09.652412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.652327    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"
Oct 30 01:09:09.652412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.652375    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7" event={"ID":"a76516bd-ba32-46c7-8a0f-878527bc1633","Type":"ContainerDied","Data":"b60a46fc27aed9ae44f59c87e1e17cb67e928649fe13f7bb983aef57a8c065b0"}
Oct 30 01:09:09.652968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.652447    5104 scope.go:122] "RemoveContainer" containerID="5d16ccf4848fc7e79b0f97af660da0fe5adce933b14a9ec6d118a6fc92272d16"
Oct 30 01:09:09.750723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.750688    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"]
Oct 30 01:09:09.755726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:09.755689    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-260268-openshift-default-d64f98dbf-6rrh7"]
Oct 30 01:09:10.586503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:10.586440    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" path="/var/lib/kubelet/pods/a76516bd-ba32-46c7-8a0f-878527bc1633/volumes"
Oct 30 01:09:13.342357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.342275    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:09:13.342807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.342355    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 01:09:13.416282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.416182    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"]
Oct 30 01:09:13.444704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.444666    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"]
Oct 30 01:09:13.707360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.707331    5104 generic.go:356] "Generic (PLEG): container finished" podID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerID="ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39" exitCode=0
Oct 30 01:09:13.707603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.707379    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" event={"ID":"d42f8ae0-21a7-446c-95f0-06b568a26d87","Type":"ContainerDied","Data":"ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39"}
Oct 30 01:09:13.707993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.707964    5104 scope.go:122] "RemoveContainer" containerID="ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39"
Oct 30 01:09:13.710380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.710345    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:13.710497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.710441    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:13.710548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:13.710509    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:14.731052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:14.731016    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" event={"ID":"d42f8ae0-21a7-446c-95f0-06b568a26d87","Type":"ContainerStarted","Data":"0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86"}
Oct 30 01:09:15.012895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.012805    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"]
Oct 30 01:09:15.014396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.014363    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy"
Oct 30 01:09:15.014594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.014578    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy"
Oct 30 01:09:15.014782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.014770    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a6a12606-b93f-4352-927a-2cdf6b5a70f5" containerName="revoke-keys"
Oct 30 01:09:15.014867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.014856    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a12606-b93f-4352-927a-2cdf6b5a70f5" containerName="revoke-keys"
Oct 30 01:09:15.015166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.015148    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy"
Oct 30 01:09:15.015302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.015291    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a12606-b93f-4352-927a-2cdf6b5a70f5" containerName="revoke-keys"
Oct 30 01:09:15.015465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.015442    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy"
Oct 30 01:09:15.028364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.025740    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"]
Oct 30 01:09:15.028364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.025863    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.041041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.041011    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fb3370cb-openshift-default-dockercfg-pnjnh\""
Oct 30 01:09:15.180483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.173469    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-credential-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.180942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.180907    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.181159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.181142    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-token\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.181417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.181397    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qvf\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-kube-api-access-d5qvf\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.181619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.181601    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-podinfo\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.181807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.181792    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-envoy\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.182052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.182032    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-data\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.182278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.182263    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-certs\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.182542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.182526    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istiod-ca-cert\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.288697    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istiod-ca-cert\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.288827    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-credential-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.288885    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.288919    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-token\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.289025    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qvf\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-kube-api-access-d5qvf\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.289049    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-podinfo\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.289090    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-envoy\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.289197    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-data\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.289535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.289288    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-certs\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.290161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.289800    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-certs\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.295489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.290500    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istiod-ca-cert\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.295489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.290794    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-credential-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.295489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.291025    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.295489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.292069    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-data\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.309493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.308176    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-podinfo\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.310126 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.310099    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-envoy\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.310838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.310803    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" containerID="cri-o://2bdbf866d546f9a3176e444a724523f9cec792df37209bd655740cdca8f9868c" gracePeriod=45
Oct 30 01:09:15.313764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.313744    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-token\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.323887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.323866    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qvf\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-kube-api-access-d5qvf\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.343137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.343113    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:15.505502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.505197    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:09:15.505502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.505324    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:09:15.505502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.505369    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 01:09:15.731617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.726611    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": dial tcp 10.132.0.170:8000: connect: connection refused"
Oct 30 01:09:15.748034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.748002    5104 generic.go:356] "Generic (PLEG): container finished" podID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerID="2bdbf866d546f9a3176e444a724523f9cec792df37209bd655740cdca8f9868c" exitCode=0
Oct 30 01:09:15.748539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.748510    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" event={"ID":"fea7f268-06b4-4a01-b283-f6ee7692bdf6","Type":"ContainerDied","Data":"2bdbf866d546f9a3176e444a724523f9cec792df37209bd655740cdca8f9868c"}
Oct 30 01:09:15.915660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.915625    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"]
Oct 30 01:09:15.924225 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:09:15.924196    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ea1b08_e9bf_41ac_bd08_3b91e11b2625.slice/crio-cca0b259fe1e7f47703c6f9cfc80e336c7aef186e96233f1aeea9fa6811c5af2 WatchSource:0}: Error finding container cca0b259fe1e7f47703c6f9cfc80e336c7aef186e96233f1aeea9fa6811c5af2: Status 404 returned error can't find the container with id cca0b259fe1e7f47703c6f9cfc80e336c7aef186e96233f1aeea9fa6811c5af2
Oct 30 01:09:15.925878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.925843    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:15.925982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.925918    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:15.925982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.925957    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:15.928698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:15.928336    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:09:16.121479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.120969    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tls-certs\") pod \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") "
Oct 30 01:09:16.121479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.121110    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-dshm\") pod \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") "
Oct 30 01:09:16.121479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.121167    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tmp-dir\") pod \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") "
Oct 30 01:09:16.121479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.121384    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-home\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-home\") pod \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") "
Oct 30 01:09:16.121854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.121493    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-model-cache\") pod \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") "
Oct 30 01:09:16.121854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.121539    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea7f268-06b4-4a01-b283-f6ee7692bdf6-kube-api-access-t8wgm\" (UniqueName: \"kubernetes.io/projected/fea7f268-06b4-4a01-b283-f6ee7692bdf6-kube-api-access-t8wgm\") pod \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\" (UID: \"fea7f268-06b4-4a01-b283-f6ee7692bdf6\") "
Oct 30 01:09:16.132032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.131942    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-model-cache" pod "fea7f268-06b4-4a01-b283-f6ee7692bdf6" (UID: "fea7f268-06b4-4a01-b283-f6ee7692bdf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:16.132209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.132066    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-home" pod "fea7f268-06b4-4a01-b283-f6ee7692bdf6" (UID: "fea7f268-06b4-4a01-b283-f6ee7692bdf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:16.132292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.132236    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tmp-dir" pod "fea7f268-06b4-4a01-b283-f6ee7692bdf6" (UID: "fea7f268-06b4-4a01-b283-f6ee7692bdf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:16.133776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.133749    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea7f268-06b4-4a01-b283-f6ee7692bdf6-kube-api-access-t8wgm" pod "fea7f268-06b4-4a01-b283-f6ee7692bdf6" (UID: "fea7f268-06b4-4a01-b283-f6ee7692bdf6"). InnerVolumeSpecName "kube-api-access-t8wgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:16.134774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.134751    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-dshm" pod "fea7f268-06b4-4a01-b283-f6ee7692bdf6" (UID: "fea7f268-06b4-4a01-b283-f6ee7692bdf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:16.135310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.135288    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tls-certs" pod "fea7f268-06b4-4a01-b283-f6ee7692bdf6" (UID: "fea7f268-06b4-4a01-b283-f6ee7692bdf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:16.224482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.224417    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:16.224793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.224703    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:16.224793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.224732    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wgm\" (UniqueName: \"kubernetes.io/projected/fea7f268-06b4-4a01-b283-f6ee7692bdf6-kube-api-access-t8wgm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:16.224793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.224746    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:16.224793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.224761    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:16.224793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.224773    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea7f268-06b4-4a01-b283-f6ee7692bdf6-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:16.511396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.510756    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:09:16.511396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.510823    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 01:09:16.759623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.759590    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"
Oct 30 01:09:16.760094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.759623    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4" event={"ID":"fea7f268-06b4-4a01-b283-f6ee7692bdf6","Type":"ContainerDied","Data":"a74552f6d98ea8e54becbb14df449fa09f7220489ddf086b5a0a83d1b219a4a4"}
Oct 30 01:09:16.760094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.759682    5104 scope.go:122] "RemoveContainer" containerID="2bdbf866d546f9a3176e444a724523f9cec792df37209bd655740cdca8f9868c"
Oct 30 01:09:16.762010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.761982    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" event={"ID":"91ea1b08-e9bf-41ac-bd08-3b91e11b2625","Type":"ContainerStarted","Data":"97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781"}
Oct 30 01:09:16.762135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.762039    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" event={"ID":"91ea1b08-e9bf-41ac-bd08-3b91e11b2625","Type":"ContainerStarted","Data":"cca0b259fe1e7f47703c6f9cfc80e336c7aef186e96233f1aeea9fa6811c5af2"}
Oct 30 01:09:16.827719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.827631    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podStartSLOduration=2.8276079320000003 podStartE2EDuration="2.827607932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:16.822312961 +0000 UTC m=+1598.370003457" watchObservedRunningTime="2026-10-30 01:09:16.827607932 +0000 UTC m=+1598.375298428"
Oct 30 01:09:16.870665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.870532    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"]
Oct 30 01:09:16.948573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:16.948079    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fb3370cb/e2e-lifecycle-model-fb3370cb-kserve-697fbbfd5c-j8xn4"]
Oct 30 01:09:17.344563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:17.344529    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:17.353656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:17.353620    5104 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:09:17.353927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:17.353895    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 01:09:17.497485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:17.496809    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:09:17.497485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:17.496868    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 01:09:18.343734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.343691    5104 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:09:18.344157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.343759    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 01:09:18.423481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.421534    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm" podUID="80c0c049-e530-4089-b133-00d5b8a695d2" containerName="payload-processing" containerID="cri-o://4b0d997b26827b61d5e2b13ded9f17c947e44f94ef5200407451a1fd1224d573" gracePeriod=25
Oct 30 01:09:18.445545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.445503    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp" podUID="96558580-ea47-4d9a-851c-8b2ef823a82b" containerName="payload-pre-processing" containerID="cri-o://7369f40e80d919fc09aa52a7bd70a773801cb397d2267ad3b640602aee71972f" gracePeriod=25
Oct 30 01:09:18.501481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.497620    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:09:18.501481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.497679    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 01:09:18.654128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.654022    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" path="/var/lib/kubelet/pods/fea7f268-06b4-4a01-b283-f6ee7692bdf6/volumes"
Oct 30 01:09:18.854243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.854197    5104 generic.go:356] "Generic (PLEG): container finished" podID="80c0c049-e530-4089-b133-00d5b8a695d2" containerID="4b0d997b26827b61d5e2b13ded9f17c947e44f94ef5200407451a1fd1224d573" exitCode=0
Oct 30 01:09:18.854409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.854250    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm" event={"ID":"80c0c049-e530-4089-b133-00d5b8a695d2","Type":"ContainerDied","Data":"4b0d997b26827b61d5e2b13ded9f17c947e44f94ef5200407451a1fd1224d573"}
Oct 30 01:09:18.860495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.857117    5104 generic.go:356] "Generic (PLEG): container finished" podID="96558580-ea47-4d9a-851c-8b2ef823a82b" containerID="7369f40e80d919fc09aa52a7bd70a773801cb397d2267ad3b640602aee71972f" exitCode=0
Oct 30 01:09:18.860495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:18.857205    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp" event={"ID":"96558580-ea47-4d9a-851c-8b2ef823a82b","Type":"ContainerDied","Data":"7369f40e80d919fc09aa52a7bd70a773801cb397d2267ad3b640602aee71972f"}
Oct 30 01:09:19.348329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.345229    5104 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:09:19.348329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.345294    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 01:09:19.395755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.395718    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:09:19.413708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.413675    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:09:19.507875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.497637    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:09:19.507875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.501540    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 01:09:19.546906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.546809    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c0c049-e530-4089-b133-00d5b8a695d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c0c049-e530-4089-b133-00d5b8a695d2-plugins-config-volume\") pod \"80c0c049-e530-4089-b133-00d5b8a695d2\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") "
Oct 30 01:09:19.547069 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.546913    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c0c049-e530-4089-b133-00d5b8a695d2-kube-api-access-c98pq\" (UniqueName: \"kubernetes.io/projected/80c0c049-e530-4089-b133-00d5b8a695d2-kube-api-access-c98pq\") pod \"80c0c049-e530-4089-b133-00d5b8a695d2\" (UID: \"80c0c049-e530-4089-b133-00d5b8a695d2\") "
Oct 30 01:09:19.547116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.547081    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96558580-ea47-4d9a-851c-8b2ef823a82b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96558580-ea47-4d9a-851c-8b2ef823a82b-plugins-config-volume\") pod \"96558580-ea47-4d9a-851c-8b2ef823a82b\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") "
Oct 30 01:09:19.547152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.547127    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96558580-ea47-4d9a-851c-8b2ef823a82b-kube-api-access-s4frg\" (UniqueName: \"kubernetes.io/projected/96558580-ea47-4d9a-851c-8b2ef823a82b-kube-api-access-s4frg\") pod \"96558580-ea47-4d9a-851c-8b2ef823a82b\" (UID: \"96558580-ea47-4d9a-851c-8b2ef823a82b\") "
Oct 30 01:09:19.547862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.547817    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96558580-ea47-4d9a-851c-8b2ef823a82b-plugins-config-volume" pod "96558580-ea47-4d9a-851c-8b2ef823a82b" (UID: "96558580-ea47-4d9a-851c-8b2ef823a82b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:19.548176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.548148    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c0c049-e530-4089-b133-00d5b8a695d2-plugins-config-volume" pod "80c0c049-e530-4089-b133-00d5b8a695d2" (UID: "80c0c049-e530-4089-b133-00d5b8a695d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:19.549740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.549711    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c0c049-e530-4089-b133-00d5b8a695d2-kube-api-access-c98pq" pod "80c0c049-e530-4089-b133-00d5b8a695d2" (UID: "80c0c049-e530-4089-b133-00d5b8a695d2"). InnerVolumeSpecName "kube-api-access-c98pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:19.550045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.550024    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96558580-ea47-4d9a-851c-8b2ef823a82b-kube-api-access-s4frg" pod "96558580-ea47-4d9a-851c-8b2ef823a82b" (UID: "96558580-ea47-4d9a-851c-8b2ef823a82b"). InnerVolumeSpecName "kube-api-access-s4frg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:19.649499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.649441    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96558580-ea47-4d9a-851c-8b2ef823a82b-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.649499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.649502    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4frg\" (UniqueName: \"kubernetes.io/projected/96558580-ea47-4d9a-851c-8b2ef823a82b-kube-api-access-s4frg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.649718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.649518    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c0c049-e530-4089-b133-00d5b8a695d2-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.649718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.649533    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98pq\" (UniqueName: \"kubernetes.io/projected/80c0c049-e530-4089-b133-00d5b8a695d2-kube-api-access-c98pq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.870597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.868336    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"
Oct 30 01:09:19.870597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.868387    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm" event={"ID":"80c0c049-e530-4089-b133-00d5b8a695d2","Type":"ContainerDied","Data":"a8c1e240b7635f54378fa5a65e23e789f8c11bd2ac072701d3756378b2dcd9fa"}
Oct 30 01:09:19.870597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.868485    5104 scope.go:122] "RemoveContainer" containerID="4b0d997b26827b61d5e2b13ded9f17c947e44f94ef5200407451a1fd1224d573"
Oct 30 01:09:19.899478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.892143    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp" event={"ID":"96558580-ea47-4d9a-851c-8b2ef823a82b","Type":"ContainerDied","Data":"ed6a492297ddc7d6af0c213dbcc0925aa7ec5d301e694ea77caa40dbbfbb38b9"}
Oct 30 01:09:19.899478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.892266    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"
Oct 30 01:09:19.956362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.956337    5104 scope.go:122] "RemoveContainer" containerID="7369f40e80d919fc09aa52a7bd70a773801cb397d2267ad3b640602aee71972f"
Oct 30 01:09:19.995487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:19.992717    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"]
Oct 30 01:09:20.009482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.001855    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07410c-7598fb6b98-8r4jm"]
Oct 30 01:09:20.033243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.031100    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"]
Oct 30 01:09:20.038276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.038244    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07410c-67d6df4667-nwwnp"]
Oct 30 01:09:20.042935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.042904    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"]
Oct 30 01:09:20.046994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.046972    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159cce8-qsqln"]
Oct 30 01:09:20.346530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.345573    5104 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:09:20.346530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.345634    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 01:09:20.498326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.497852    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:09:20.498326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.497915    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 01:09:20.664529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.662755    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c0c049-e530-4089-b133-00d5b8a695d2" path="/var/lib/kubelet/pods/80c0c049-e530-4089-b133-00d5b8a695d2/volumes"
Oct 30 01:09:20.664529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.663382    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96558580-ea47-4d9a-851c-8b2ef823a82b" path="/var/lib/kubelet/pods/96558580-ea47-4d9a-851c-8b2ef823a82b/volumes"
Oct 30 01:09:20.664529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:20.664014    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cb0863-578b-4ac3-b13e-33cb64aca759" path="/var/lib/kubelet/pods/b0cb0863-578b-4ac3-b13e-33cb64aca759/volumes"
Oct 30 01:09:21.343861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:21.343821    5104 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:09:21.344056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:21.343891    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 01:09:21.565983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:21.565828    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:09:21.567089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:21.566587    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:09:21.569901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:21.567398    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:09:22.344447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.344407    5104 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:09:22.344670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.344509    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 01:09:22.835481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.820204    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"]
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.837337    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main"
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.839561    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main"
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.839615    5104 cpu_manager.go:414] "containerMap: removing container" podUID="80c0c049-e530-4089-b133-00d5b8a695d2" containerName="payload-processing"
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.839625    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c0c049-e530-4089-b133-00d5b8a695d2" containerName="payload-processing"
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.839678    5104 cpu_manager.go:414] "containerMap: removing container" podUID="96558580-ea47-4d9a-851c-8b2ef823a82b" containerName="payload-pre-processing"
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.839686    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96558580-ea47-4d9a-851c-8b2ef823a82b" containerName="payload-pre-processing"
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.839815    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy"
Oct 30 01:09:22.840067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.839829    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76516bd-ba32-46c7-8a0f-878527bc1633" containerName="istio-proxy"
Oct 30 01:09:22.840639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.840102    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea7f268-06b4-4a01-b283-f6ee7692bdf6" containerName="main"
Oct 30 01:09:22.840639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.840120    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c0c049-e530-4089-b133-00d5b8a695d2" containerName="payload-processing"
Oct 30 01:09:22.840639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.840139    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="96558580-ea47-4d9a-851c-8b2ef823a82b" containerName="payload-pre-processing"
Oct 30 01:09:22.869567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.861108    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"]
Oct 30 01:09:22.869567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.861576    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:22.885356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.881362    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:22.896499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.890844    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:22.915727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.915693    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:22.915946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.915931    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:09:22.922904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.922869    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzg4n\" (UniqueName: \"kubernetes.io/projected/b17bc456-2e63-42df-954a-88c26f991768-kube-api-access-xzg4n\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:22.923188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.923167    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17bc456-2e63-42df-954a-88c26f991768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:22.923399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:22.923380    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc456-2e63-42df-954a-88c26f991768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.025071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.025032    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzg4n\" (UniqueName: \"kubernetes.io/projected/b17bc456-2e63-42df-954a-88c26f991768-kube-api-access-xzg4n\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.025279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.025123    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17bc456-2e63-42df-954a-88c26f991768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.025279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.025215    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc456-2e63-42df-954a-88c26f991768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.026234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.026209    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc456-2e63-42df-954a-88c26f991768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.028349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.028326    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17bc456-2e63-42df-954a-88c26f991768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.046358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.046312    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzg4n\" (UniqueName: \"kubernetes.io/projected/b17bc456-2e63-42df-954a-88c26f991768-kube-api-access-xzg4n\") pod \"maas-api-revoke-sub-5534addc-jqhjh\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.226317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.226266    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:23.445814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.445779    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:23.447393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.446558    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:23.448506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.447743    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:23.541689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:23.541569    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"]
Oct 30 01:09:24.044199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:24.044161    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh" event={"ID":"b17bc456-2e63-42df-954a-88c26f991768","Type":"ContainerStarted","Data":"5f7d6a0f95f93b4eea369fc954e1afbb79221e86a2c6fd5d70e3bd364eaf642b"}
Oct 30 01:09:24.044595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:24.044225    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh" event={"ID":"b17bc456-2e63-42df-954a-88c26f991768","Type":"ContainerStarted","Data":"8693a95bb5f0e4659f6e0d9de7846b881d823a9cd562e75608d939e64bec6484"}
Oct 30 01:09:24.084418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:24.084304    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh" podStartSLOduration=2.084282428 podStartE2EDuration="2.084282428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:24.082723058 +0000 UTC m=+1605.630413553" watchObservedRunningTime="2026-10-30 01:09:24.084282428 +0000 UTC m=+1605.631972925"
Oct 30 01:09:25.068812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:25.068781    5104 generic.go:356] "Generic (PLEG): container finished" podID="b17bc456-2e63-42df-954a-88c26f991768" containerID="5f7d6a0f95f93b4eea369fc954e1afbb79221e86a2c6fd5d70e3bd364eaf642b" exitCode=0
Oct 30 01:09:25.069317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:25.068865    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh" event={"ID":"b17bc456-2e63-42df-954a-88c26f991768","Type":"ContainerDied","Data":"5f7d6a0f95f93b4eea369fc954e1afbb79221e86a2c6fd5d70e3bd364eaf642b"}
Oct 30 01:09:26.284631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.284603    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:26.406127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.406036    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b17bc456-2e63-42df-954a-88c26f991768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17bc456-2e63-42df-954a-88c26f991768-maas-api-xks-ca\") pod \"b17bc456-2e63-42df-954a-88c26f991768\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") "
Oct 30 01:09:26.406437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.406412    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17bc456-2e63-42df-954a-88c26f991768-kube-api-access-xzg4n\" (UniqueName: \"kubernetes.io/projected/b17bc456-2e63-42df-954a-88c26f991768-kube-api-access-xzg4n\") pod \"b17bc456-2e63-42df-954a-88c26f991768\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") "
Oct 30 01:09:26.406966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.406942    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17bc456-2e63-42df-954a-88c26f991768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc456-2e63-42df-954a-88c26f991768-maas-api-openshift-ca\") pod \"b17bc456-2e63-42df-954a-88c26f991768\" (UID: \"b17bc456-2e63-42df-954a-88c26f991768\") "
Oct 30 01:09:26.423754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.423711    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17bc456-2e63-42df-954a-88c26f991768-maas-api-openshift-ca" pod "b17bc456-2e63-42df-954a-88c26f991768" (UID: "b17bc456-2e63-42df-954a-88c26f991768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:26.667625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.591441    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc456-2e63-42df-954a-88c26f991768-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:26.685759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.663301    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17bc456-2e63-42df-954a-88c26f991768-kube-api-access-xzg4n" pod "b17bc456-2e63-42df-954a-88c26f991768" (UID: "b17bc456-2e63-42df-954a-88c26f991768"). InnerVolumeSpecName "kube-api-access-xzg4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:26.686051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.663885    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17bc456-2e63-42df-954a-88c26f991768-maas-api-xks-ca" pod "b17bc456-2e63-42df-954a-88c26f991768" (UID: "b17bc456-2e63-42df-954a-88c26f991768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:26.987549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.968348    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17bc456-2e63-42df-954a-88c26f991768-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:26.987549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:26.968389    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzg4n\" (UniqueName: \"kubernetes.io/projected/b17bc456-2e63-42df-954a-88c26f991768-kube-api-access-xzg4n\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:27.093901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.093873    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"
Oct 30 01:09:27.093901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.093884    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh" event={"ID":"b17bc456-2e63-42df-954a-88c26f991768","Type":"ContainerDied","Data":"8693a95bb5f0e4659f6e0d9de7846b881d823a9cd562e75608d939e64bec6484"}
Oct 30 01:09:27.094163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.093925    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8693a95bb5f0e4659f6e0d9de7846b881d823a9cd562e75608d939e64bec6484"
Oct 30 01:09:27.126948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.126910    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"]
Oct 30 01:09:27.131904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.131877    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-288da6-tzl2v"]
Oct 30 01:09:27.251691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.251599    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"]
Oct 30 01:09:27.253175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.253143    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b17bc456-2e63-42df-954a-88c26f991768" containerName="revoke-keys"
Oct 30 01:09:27.253355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.253342    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17bc456-2e63-42df-954a-88c26f991768" containerName="revoke-keys"
Oct 30 01:09:27.272219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.258961    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17bc456-2e63-42df-954a-88c26f991768" containerName="revoke-keys"
Oct 30 01:09:27.292533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.290671    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"]
Oct 30 01:09:27.292533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.290771    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.305827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.305801    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:27.306062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.306042    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:27.306836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.306819    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:27.306938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.306860    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:09:27.387614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.387575    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.387614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.387616    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv58c\" (UniqueName: \"kubernetes.io/projected/f697bb2a-fc15-477c-9130-9f93822e515a-kube-api-access-hv58c\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.387920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.387823    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.489448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.489409    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.489712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.489507    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv58c\" (UniqueName: \"kubernetes.io/projected/f697bb2a-fc15-477c-9130-9f93822e515a-kube-api-access-hv58c\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.489712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.489679    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.490205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.490183    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.492070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.492050    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.511217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.511148    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv58c\" (UniqueName: \"kubernetes.io/projected/f697bb2a-fc15-477c-9130-9f93822e515a-kube-api-access-hv58c\") pod \"maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.612904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.612863    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:27.918522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:27.918493    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"]
Oct 30 01:09:27.919079 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:09:27.919046    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf697bb2a_fc15_477c_9130_9f93822e515a.slice/crio-5edbc898037fe4fb0cc7910f41372e51c5691eb52abfb287d4ac82d0ce094544 WatchSource:0}: Error finding container 5edbc898037fe4fb0cc7910f41372e51c5691eb52abfb287d4ac82d0ce094544: Status 404 returned error can't find the container with id 5edbc898037fe4fb0cc7910f41372e51c5691eb52abfb287d4ac82d0ce094544
Oct 30 01:09:28.122296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:28.122230    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp" event={"ID":"f697bb2a-fc15-477c-9130-9f93822e515a","Type":"ContainerStarted","Data":"5edbc898037fe4fb0cc7910f41372e51c5691eb52abfb287d4ac82d0ce094544"}
Oct 30 01:09:28.582964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:28.582882    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a0f230-08dc-496e-b508-be4c7e908d42" path="/var/lib/kubelet/pods/a3a0f230-08dc-496e-b508-be4c7e908d42/volumes"
Oct 30 01:09:29.156386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:29.156290    5104 generic.go:356] "Generic (PLEG): container finished" podID="f697bb2a-fc15-477c-9130-9f93822e515a" containerID="35ea32023c768f2d524959e256ebcfca68f0b7535f9df352b3b775343bfd82be" exitCode=0
Oct 30 01:09:29.156581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:29.156421    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp" event={"ID":"f697bb2a-fc15-477c-9130-9f93822e515a","Type":"ContainerDied","Data":"35ea32023c768f2d524959e256ebcfca68f0b7535f9df352b3b775343bfd82be"}
Oct 30 01:09:30.347236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.347208    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:30.438123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.438093    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f697bb2a-fc15-477c-9130-9f93822e515a-kube-api-access-hv58c\" (UniqueName: \"kubernetes.io/projected/f697bb2a-fc15-477c-9130-9f93822e515a-kube-api-access-hv58c\") pod \"f697bb2a-fc15-477c-9130-9f93822e515a\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") "
Oct 30 01:09:30.438307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.438223    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-openshift-ca\") pod \"f697bb2a-fc15-477c-9130-9f93822e515a\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") "
Oct 30 01:09:30.438307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.438272    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-xks-ca\") pod \"f697bb2a-fc15-477c-9130-9f93822e515a\" (UID: \"f697bb2a-fc15-477c-9130-9f93822e515a\") "
Oct 30 01:09:30.438764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.438732    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-openshift-ca" pod "f697bb2a-fc15-477c-9130-9f93822e515a" (UID: "f697bb2a-fc15-477c-9130-9f93822e515a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:30.439241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.439214    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:30.440958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.440927    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-xks-ca" pod "f697bb2a-fc15-477c-9130-9f93822e515a" (UID: "f697bb2a-fc15-477c-9130-9f93822e515a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:30.441268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.441234    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f697bb2a-fc15-477c-9130-9f93822e515a-kube-api-access-hv58c" pod "f697bb2a-fc15-477c-9130-9f93822e515a" (UID: "f697bb2a-fc15-477c-9130-9f93822e515a"). InnerVolumeSpecName "kube-api-access-hv58c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:30.541190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.541158    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv58c\" (UniqueName: \"kubernetes.io/projected/f697bb2a-fc15-477c-9130-9f93822e515a-kube-api-access-hv58c\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:30.541190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:30.541189    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f697bb2a-fc15-477c-9130-9f93822e515a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:31.177995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:31.177962    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"
Oct 30 01:09:31.178199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:31.177999    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp" event={"ID":"f697bb2a-fc15-477c-9130-9f93822e515a","Type":"ContainerDied","Data":"5edbc898037fe4fb0cc7910f41372e51c5691eb52abfb287d4ac82d0ce094544"}
Oct 30 01:09:31.178199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:31.178063    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5edbc898037fe4fb0cc7910f41372e51c5691eb52abfb287d4ac82d0ce094544"
Oct 30 01:09:37.313425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:37.313390    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"]
Oct 30 01:09:37.314275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:37.314237    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy" containerID="cri-o://17e674fa19e113a3d15711509f0a22ec7d5a8de969feaf13852afc89dab52340" gracePeriod=30
Oct 30 01:09:37.346763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:37.346719    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"]
Oct 30 01:09:37.347059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:37.347023    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy" containerID="cri-o://4ea87c5023e101032fff0f15e68d2300458e6bc42adee8cc39aff25d3af6e48a" gracePeriod=30
Oct 30 01:09:37.355302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:37.355269    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"]
Oct 30 01:09:37.355557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:37.355533    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy" containerID="cri-o://e8fdb66e1de702f9768e608da81cdfc85eaef9a21984c72edbb16f46bd9cf029" gracePeriod=30
Oct 30 01:09:39.106334 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:09:39.106291    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:09:41.795869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.795831    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"]
Oct 30 01:09:41.797167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.797143    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f697bb2a-fc15-477c-9130-9f93822e515a" containerName="revoke-keys"
Oct 30 01:09:41.797237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.797170    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f697bb2a-fc15-477c-9130-9f93822e515a" containerName="revoke-keys"
Oct 30 01:09:41.797365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.797352    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f697bb2a-fc15-477c-9130-9f93822e515a" containerName="revoke-keys"
Oct 30 01:09:41.802273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.802246    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:41.815201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.815172    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:41.815345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.815228    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86hn\" (UniqueName: \"kubernetes.io/projected/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-kube-api-access-t86hn\") pod \"payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:41.831341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.831305    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-52f1cb\""
Oct 30 01:09:41.831529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.831430    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-52f1cb-dockercfg-phfnw\""
Oct 30 01:09:41.841365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.841332    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"]
Oct 30 01:09:41.956297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.947654    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:41.956297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.947744    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86hn\" (UniqueName: \"kubernetes.io/projected/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-kube-api-access-t86hn\") pod \"payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:41.956297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:41.949329    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:42.104539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.104398    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86hn\" (UniqueName: \"kubernetes.io/projected/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-kube-api-access-t86hn\") pod \"payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:42.216486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.214186    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:42.285843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.285812    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"]
Oct 30 01:09:42.301739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.301709    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"]
Oct 30 01:09:42.302090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.302053    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.317230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.316692    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f33366-e2a3-4e2b-86a5-4924e65d3b69-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.317230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.316757    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzspv\" (UniqueName: \"kubernetes.io/projected/23f33366-e2a3-4e2b-86a5-4924e65d3b69-kube-api-access-qzspv\") pod \"payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.420759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.419702    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f33366-e2a3-4e2b-86a5-4924e65d3b69-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.420759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.419756    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzspv\" (UniqueName: \"kubernetes.io/projected/23f33366-e2a3-4e2b-86a5-4924e65d3b69-kube-api-access-qzspv\") pod \"payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.421232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.421173    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f33366-e2a3-4e2b-86a5-4924e65d3b69-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.439891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.439826    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzspv\" (UniqueName: \"kubernetes.io/projected/23f33366-e2a3-4e2b-86a5-4924e65d3b69-kube-api-access-qzspv\") pod \"payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.607318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.603731    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:42.608997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.608913    5104 generic.go:356] "Generic (PLEG): container finished" podID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerID="4ea87c5023e101032fff0f15e68d2300458e6bc42adee8cc39aff25d3af6e48a" exitCode=0
Oct 30 01:09:42.615514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.614922    5104 generic.go:356] "Generic (PLEG): container finished" podID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerID="e8fdb66e1de702f9768e608da81cdfc85eaef9a21984c72edbb16f46bd9cf029" exitCode=0
Oct 30 01:09:42.620179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.620118    5104 generic.go:356] "Generic (PLEG): container finished" podID="61231598-2fc8-4806-97a9-fc167d024f3c" containerID="17e674fa19e113a3d15711509f0a22ec7d5a8de969feaf13852afc89dab52340" exitCode=0
Oct 30 01:09:42.663500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.653629    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" event={"ID":"6e8481aa-9575-4f53-a63c-52c5e7826750","Type":"ContainerDied","Data":"4ea87c5023e101032fff0f15e68d2300458e6bc42adee8cc39aff25d3af6e48a"}
Oct 30 01:09:42.663500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.654369    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" event={"ID":"fd059c20-b2e8-41f5-81d8-19b831922f9e","Type":"ContainerDied","Data":"e8fdb66e1de702f9768e608da81cdfc85eaef9a21984c72edbb16f46bd9cf029"}
Oct 30 01:09:42.663500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.654417    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" event={"ID":"61231598-2fc8-4806-97a9-fc167d024f3c","Type":"ContainerDied","Data":"17e674fa19e113a3d15711509f0a22ec7d5a8de969feaf13852afc89dab52340"}
Oct 30 01:09:42.671666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.670167    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"]
Oct 30 01:09:42.843759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.843321    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:09:42.931685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.931659    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:09:42.933676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.933652    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61231598-2fc8-4806-97a9-fc167d024f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61231598-2fc8-4806-97a9-fc167d024f3c-istio-podinfo\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.933843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.933785    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-data\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.933903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.933843    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-socket\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.933979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.933964    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-credential-socket\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.934039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934019    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61231598-2fc8-4806-97a9-fc167d024f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61231598-2fc8-4806-97a9-fc167d024f3c-istiod-ca-cert\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.934100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934073    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-kube-api-access-dj75z\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-kube-api-access-dj75z\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.934149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934092    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-data" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:42.934149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934113    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-socket" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:42.934245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934196    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-certs\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.934245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934209    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-credential-socket" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:42.934356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934289    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-envoy\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.934356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934337    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-istio-token\") pod \"61231598-2fc8-4806-97a9-fc167d024f3c\" (UID: \"61231598-2fc8-4806-97a9-fc167d024f3c\") "
Oct 30 01:09:42.934494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934422    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-certs" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:42.934554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.934495    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61231598-2fc8-4806-97a9-fc167d024f3c-istiod-ca-cert" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:42.935147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.935119    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.935253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.935150    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.935253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.935165    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61231598-2fc8-4806-97a9-fc167d024f3c-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.935253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.935177    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.935253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.935190    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:42.935977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.935959    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:09:42.936664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.936642    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61231598-2fc8-4806-97a9-fc167d024f3c-istio-podinfo" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:42.936790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.936772    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-envoy" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:42.937145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.937128    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-istio-token" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:42.939061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.939035    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-kube-api-access-dj75z" pod "61231598-2fc8-4806-97a9-fc167d024f3c" (UID: "61231598-2fc8-4806-97a9-fc167d024f3c"). InnerVolumeSpecName "kube-api-access-dj75z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:42.969600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:42.969572    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"]
Oct 30 01:09:42.975529 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:09:42.972432    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f33366_e2a3_4e2b_86a5_4924e65d3b69.slice/crio-fa6e96c09a1ba11ebb9f051bd498598234a10587ed58e5bea8411477e0644c51 WatchSource:0}: Error finding container fa6e96c09a1ba11ebb9f051bd498598234a10587ed58e5bea8411477e0644c51: Status 404 returned error can't find the container with id fa6e96c09a1ba11ebb9f051bd498598234a10587ed58e5bea8411477e0644c51
Oct 30 01:09:43.036303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036274    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8481aa-9575-4f53-a63c-52c5e7826750-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8481aa-9575-4f53-a63c-52c5e7826750-istiod-ca-cert\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.036512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036325    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-envoy\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.036512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036345    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-certs\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.036512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036368    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-credential-socket\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.036512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036467    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-envoy\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036523    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-token\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-token\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036549    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-token\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-token\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036565    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-data\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036582    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd059c20-b2e8-41f5-81d8-19b831922f9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd059c20-b2e8-41f5-81d8-19b831922f9e-istiod-ca-cert\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036641    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-podinfo\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036655    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8481aa-9575-4f53-a63c-52c5e7826750-istiod-ca-cert" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036670    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-podinfo\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.036759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036729    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-kube-api-access-tzl6r\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-kube-api-access-tzl6r\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.037061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036857    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-socket\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.037061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036905    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-kube-api-access-rv2hj\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-kube-api-access-rv2hj\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.037061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036959    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-certs\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.037061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.036991    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-data\") pod \"6e8481aa-9575-4f53-a63c-52c5e7826750\" (UID: \"6e8481aa-9575-4f53-a63c-52c5e7826750\") "
Oct 30 01:09:43.037278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037067    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-socket\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.037278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037102    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-credential-socket\") pod \"fd059c20-b2e8-41f5-81d8-19b831922f9e\" (UID: \"fd059c20-b2e8-41f5-81d8-19b831922f9e\") "
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037880    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8481aa-9575-4f53-a63c-52c5e7826750-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037886    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-data" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037899    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj75z\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-kube-api-access-dj75z\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037943    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61231598-2fc8-4806-97a9-fc167d024f3c-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037960    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61231598-2fc8-4806-97a9-fc167d024f3c-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.037975    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61231598-2fc8-4806-97a9-fc167d024f3c-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.038060    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-credential-socket" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.038183    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd059c20-b2e8-41f5-81d8-19b831922f9e-istiod-ca-cert" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.038681    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-certs" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.038717    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-socket" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.038786    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-credential-socket" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.038840    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-certs" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.038999    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-data" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.039121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.039088    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-socket" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.040704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.040677    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-podinfo" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:43.041283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.041262    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-kube-api-access-tzl6r" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "kube-api-access-tzl6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:43.041587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.041553    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-kube-api-access-rv2hj" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "kube-api-access-rv2hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:43.042942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.042904    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-token" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:43.043106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.043076    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-envoy" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.043450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.043417    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-envoy" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:43.043555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.043497    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-token" pod "fd059c20-b2e8-41f5-81d8-19b831922f9e" (UID: "fd059c20-b2e8-41f5-81d8-19b831922f9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:43.045095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.045071    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-podinfo" pod "6e8481aa-9575-4f53-a63c-52c5e7826750" (UID: "6e8481aa-9575-4f53-a63c-52c5e7826750"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:43.139730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139699    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139727    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139741    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzl6r\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-kube-api-access-tzl6r\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139756    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139765    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv2hj\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-kube-api-access-rv2hj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139774    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139782    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139789    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139799    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139807    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139819    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139830    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139838    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139846    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139854    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8481aa-9575-4f53-a63c-52c5e7826750-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139861    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd059c20-b2e8-41f5-81d8-19b831922f9e-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.139979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.139868    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd059c20-b2e8-41f5-81d8-19b831922f9e-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.288382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.288299    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:09:43.698333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.698299    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" event={"ID":"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d","Type":"ContainerStarted","Data":"efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065"}
Oct 30 01:09:43.721524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.720806    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:43.721524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.720895    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" event={"ID":"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d","Type":"ContainerStarted","Data":"ee79cbea9c3b52886194c748482866dacb3b43cf2a78df27a4dd407749ffb9b5"}
Oct 30 01:09:43.730789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.730388    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" event={"ID":"23f33366-e2a3-4e2b-86a5-4924e65d3b69","Type":"ContainerStarted","Data":"758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43"}
Oct 30 01:09:43.730789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.730452    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" event={"ID":"23f33366-e2a3-4e2b-86a5-4924e65d3b69","Type":"ContainerStarted","Data":"fa6e96c09a1ba11ebb9f051bd498598234a10587ed58e5bea8411477e0644c51"}
Oct 30 01:09:43.735497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.731502    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:43.735497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.732912    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"
Oct 30 01:09:43.735497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.732924    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz" event={"ID":"61231598-2fc8-4806-97a9-fc167d024f3c","Type":"ContainerDied","Data":"ed5c6daba1e6fae60de27413366519de005609392701d5eb2c62976003a1c134"}
Oct 30 01:09:43.735497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.732971    5104 scope.go:122] "RemoveContainer" containerID="17e674fa19e113a3d15711509f0a22ec7d5a8de969feaf13852afc89dab52340"
Oct 30 01:09:43.736225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.736199    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg" event={"ID":"6e8481aa-9575-4f53-a63c-52c5e7826750","Type":"ContainerDied","Data":"b18966896e2fafda01466be08c7963dc61cc733ae851fa19bae2085ac00df27b"}
Oct 30 01:09:43.736367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.736351    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"
Oct 30 01:09:43.739031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.738996    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp" event={"ID":"fd059c20-b2e8-41f5-81d8-19b831922f9e","Type":"ContainerDied","Data":"63c0dc60eb03e417bd58d3fa03d8d6d64f712b8198190ee3ff0774c98371d3a1"}
Oct 30 01:09:43.739336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.739300    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"
Oct 30 01:09:43.750757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.750735    5104 scope.go:122] "RemoveContainer" containerID="4ea87c5023e101032fff0f15e68d2300458e6bc42adee8cc39aff25d3af6e48a"
Oct 30 01:09:43.764323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.764269    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" podStartSLOduration=2.764252411 podStartE2EDuration="2.764252411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:43.752922952 +0000 UTC m=+1625.300613450" watchObservedRunningTime="2026-10-30 01:09:43.764252411 +0000 UTC m=+1625.311942908"
Oct 30 01:09:43.766442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.766424    5104 scope.go:122] "RemoveContainer" containerID="e8fdb66e1de702f9768e608da81cdfc85eaef9a21984c72edbb16f46bd9cf029"
Oct 30 01:09:43.804549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.804450    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" podStartSLOduration=2.804432845 podStartE2EDuration="2.804432845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:43.802774122 +0000 UTC m=+1625.350464618" watchObservedRunningTime="2026-10-30 01:09:43.804432845 +0000 UTC m=+1625.352123374"
Oct 30 01:09:43.844957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.844922    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"]
Oct 30 01:09:43.855575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.855540    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-ldmwz"]
Oct 30 01:09:43.898386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.898352    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"]
Oct 30 01:09:43.905085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.905053    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-qtsxg"]
Oct 30 01:09:43.920696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.920669    5104 scope.go:122] "RemoveContainer" containerID="ee739a4cdf6031bb005574cc760d32066df5d9622b262eb15e4f49255b3c54df"
Oct 30 01:09:43.935791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.935769    5104 scope.go:122] "RemoveContainer" containerID="cf4c5e99e690afc3aca7139854f8fe8f1df9c79f33b55c17e9c550a5173e2f0e"
Oct 30 01:09:43.943122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.943096    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"]
Oct 30 01:09:43.949159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.949074    5104 scope.go:122] "RemoveContainer" containerID="73285bc54a5a17c3c5f30afa8483452e1346c0635e434fb40fa3f49022313a9c"
Oct 30 01:09:43.951965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.951941    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6wplp"]
Oct 30 01:09:43.960636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.960618    5104 scope.go:122] "RemoveContainer" containerID="b3b9230f2548e8a950e0fc08c296b0a9a54cf310d1826aa734df796d8fd683e6"
Oct 30 01:09:43.972419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:43.972396    5104 scope.go:122] "RemoveContainer" containerID="653ece8b3b7e713cbfb2393574fb77065407ad122c661f198fa5df82760ce0ff"
Oct 30 01:09:44.580981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:44.580902    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" path="/var/lib/kubelet/pods/61231598-2fc8-4806-97a9-fc167d024f3c/volumes"
Oct 30 01:09:44.581557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:44.581534    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" path="/var/lib/kubelet/pods/6e8481aa-9575-4f53-a63c-52c5e7826750/volumes"
Oct 30 01:09:44.582094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:44.582074    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" path="/var/lib/kubelet/pods/fd059c20-b2e8-41f5-81d8-19b831922f9e/volumes"
Oct 30 01:09:46.236765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:46.236727    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"]
Oct 30 01:09:46.237174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:46.236987    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy" containerID="cri-o://97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781" gracePeriod=30
Oct 30 01:09:47.814378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.814339    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"]
Oct 30 01:09:47.815600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815578    5104 cpu_manager.go:414] "containerMap: removing container" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy"
Oct 30 01:09:47.815673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815602    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy"
Oct 30 01:09:47.815673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815657    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy"
Oct 30 01:09:47.815673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815666    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy"
Oct 30 01:09:47.815778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815700    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy"
Oct 30 01:09:47.815778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815708    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy"
Oct 30 01:09:47.815936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815924    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="61231598-2fc8-4806-97a9-fc167d024f3c" containerName="istio-proxy"
Oct 30 01:09:47.815978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815947    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd059c20-b2e8-41f5-81d8-19b831922f9e" containerName="istio-proxy"
Oct 30 01:09:47.815978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.815962    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8481aa-9575-4f53-a63c-52c5e7826750" containerName="istio-proxy"
Oct 30 01:09:47.823620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.823590    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:47.839860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.837085    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"]
Oct 30 01:09:47.841578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.841548    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:47.841738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.841560    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:47.841738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.841675    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:09:47.851311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.851283    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:47.943404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.943363    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:47.943847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.943816    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:47.944409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:47.944383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97djs\" (UniqueName: \"kubernetes.io/projected/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-kube-api-access-97djs\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.045996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.045961    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.046207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.046055    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97djs\" (UniqueName: \"kubernetes.io/projected/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-kube-api-access-97djs\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.046337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.046315    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.047022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.046994    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.048443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.048418    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.065138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.065065    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97djs\" (UniqueName: \"kubernetes.io/projected/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-kube-api-access-97djs\") pod \"maas-api-revoke-sub-54d81291-rshdx\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.144165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.144125    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:48.463800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.458140    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"]
Oct 30 01:09:48.851897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:48.851850    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx" event={"ID":"aa09b2dd-244c-445d-9a55-ad46ce5b15e5","Type":"ContainerStarted","Data":"e34ce7e6282fd3236edda2795293c08a4a06e46699c13fa6d46ac304ed96fc35"}
Oct 30 01:09:49.869810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:49.869712    5104 generic.go:356] "Generic (PLEG): container finished" podID="aa09b2dd-244c-445d-9a55-ad46ce5b15e5" containerID="f4e4eb4d8964614a5c2c83cbb8b45db66c58ba71df4030abf95b9c7894678f05" exitCode=0
Oct 30 01:09:49.870292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:49.870202    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx" event={"ID":"aa09b2dd-244c-445d-9a55-ad46ce5b15e5","Type":"ContainerDied","Data":"f4e4eb4d8964614a5c2c83cbb8b45db66c58ba71df4030abf95b9c7894678f05"}
Oct 30 01:09:51.127413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.127382    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:51.228061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.228027    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-openshift-ca\") pod \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") "
Oct 30 01:09:51.228250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.228155    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-kube-api-access-97djs\" (UniqueName: \"kubernetes.io/projected/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-kube-api-access-97djs\") pod \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") "
Oct 30 01:09:51.228250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.228193    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-xks-ca\") pod \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\" (UID: \"aa09b2dd-244c-445d-9a55-ad46ce5b15e5\") "
Oct 30 01:09:51.228426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.228398    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-openshift-ca" pod "aa09b2dd-244c-445d-9a55-ad46ce5b15e5" (UID: "aa09b2dd-244c-445d-9a55-ad46ce5b15e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:51.228994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.228965    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.230429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.230404    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-kube-api-access-97djs" pod "aa09b2dd-244c-445d-9a55-ad46ce5b15e5" (UID: "aa09b2dd-244c-445d-9a55-ad46ce5b15e5"). InnerVolumeSpecName "kube-api-access-97djs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:51.230733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.230706    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-xks-ca" pod "aa09b2dd-244c-445d-9a55-ad46ce5b15e5" (UID: "aa09b2dd-244c-445d-9a55-ad46ce5b15e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:51.330799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.330741    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97djs\" (UniqueName: \"kubernetes.io/projected/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-kube-api-access-97djs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.330799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.330790    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa09b2dd-244c-445d-9a55-ad46ce5b15e5-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.520439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.520406    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:51.639286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639248    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-kube-api-access-d5qvf\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-kube-api-access-d5qvf\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.639510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639330    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-data\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.639510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639354    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-certs\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.639510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639377    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-envoy\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.639510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639439    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istiod-ca-cert\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.639782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639747    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-data" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:51.639838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639771    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-certs" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:51.639955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.639933    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istiod-ca-cert" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:51.640098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.640060    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-socket\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.640927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.640891    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-socket" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:51.641119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.641099    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-credential-socket\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.641358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.641337    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-credential-socket" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:51.641711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.641690    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-token\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-token\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.641855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.641751    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-podinfo\") pod \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\" (UID: \"91ea1b08-e9bf-41ac-bd08-3b91e11b2625\") "
Oct 30 01:09:51.642242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.642219    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-envoy" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:51.642841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.642817    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.642944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.642845    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.642944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.642860    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.642944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.642872    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.642944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.642885    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.642944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.642898    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.644733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.644702    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-token" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:51.645343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.645314    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-podinfo" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:51.651657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.651630    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-kube-api-access-d5qvf" pod "91ea1b08-e9bf-41ac-bd08-3b91e11b2625" (UID: "91ea1b08-e9bf-41ac-bd08-3b91e11b2625"). InnerVolumeSpecName "kube-api-access-d5qvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:51.744212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.744178    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.744212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.744210    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.744409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.744226    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qvf\" (UniqueName: \"kubernetes.io/projected/91ea1b08-e9bf-41ac-bd08-3b91e11b2625-kube-api-access-d5qvf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:09:51.893828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.893732    5104 generic.go:356] "Generic (PLEG): container finished" podID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerID="97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781" exitCode=0
Oct 30 01:09:51.893828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.893793    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"
Oct 30 01:09:51.894073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.893820    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" event={"ID":"91ea1b08-e9bf-41ac-bd08-3b91e11b2625","Type":"ContainerDied","Data":"97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781"}
Oct 30 01:09:51.894073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.893877    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm" event={"ID":"91ea1b08-e9bf-41ac-bd08-3b91e11b2625","Type":"ContainerDied","Data":"cca0b259fe1e7f47703c6f9cfc80e336c7aef186e96233f1aeea9fa6811c5af2"}
Oct 30 01:09:51.894073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.893905    5104 scope.go:122] "RemoveContainer" containerID="97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781"
Oct 30 01:09:51.904480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.904447    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"
Oct 30 01:09:51.905555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.904551    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx" event={"ID":"aa09b2dd-244c-445d-9a55-ad46ce5b15e5","Type":"ContainerDied","Data":"e34ce7e6282fd3236edda2795293c08a4a06e46699c13fa6d46ac304ed96fc35"}
Oct 30 01:09:51.905730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.905708    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34ce7e6282fd3236edda2795293c08a4a06e46699c13fa6d46ac304ed96fc35"
Oct 30 01:09:51.909448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.909428    5104 scope.go:122] "RemoveContainer" containerID="97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781"
Oct 30 01:09:51.909771 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:09:51.909745    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781\": container with ID starting with 97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781 not found: ID does not exist" containerID="97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781"
Oct 30 01:09:51.909851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.909784    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781"} err="failed to get container status \"97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781\": rpc error: code = NotFound desc = could not find container \"97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781\": container with ID starting with 97be3c271d29ca90769588da90dc8eb510bb449f6fad11bae7b7a33fe8db9781 not found: ID does not exist"
Oct 30 01:09:51.946208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.946177    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"]
Oct 30 01:09:51.957939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:51.957912    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-pzhwm"]
Oct 30 01:09:52.583798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.583760    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" path="/var/lib/kubelet/pods/91ea1b08-e9bf-41ac-bd08-3b91e11b2625/volumes"
Oct 30 01:09:52.830137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.830104    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"]
Oct 30 01:09:52.830937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.830920    5104 cpu_manager.go:414] "containerMap: removing container" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy"
Oct 30 01:09:52.831010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.830938    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy"
Oct 30 01:09:52.831010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.830976    5104 cpu_manager.go:414] "containerMap: removing container" podUID="aa09b2dd-244c-445d-9a55-ad46ce5b15e5" containerName="revoke-keys"
Oct 30 01:09:52.831010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.830981    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa09b2dd-244c-445d-9a55-ad46ce5b15e5" containerName="revoke-keys"
Oct 30 01:09:52.831110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.831091    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ea1b08-e9bf-41ac-bd08-3b91e11b2625" containerName="istio-proxy"
Oct 30 01:09:52.831110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.831106    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa09b2dd-244c-445d-9a55-ad46ce5b15e5" containerName="revoke-keys"
Oct 30 01:09:52.836306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.836249    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:52.850647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.850624    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e92dab\""
Oct 30 01:09:52.854921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.854889    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"]
Oct 30 01:09:52.866969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.866935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv7gp\" (UniqueName: \"kubernetes.io/projected/b02c7785-9248-485d-8d94-23290c4f6c0d-kube-api-access-sv7gp\") pod \"payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:52.867394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.867367    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02c7785-9248-485d-8d94-23290c4f6c0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:52.912976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.912943    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e92dab-dockercfg-vkpqc\""
Oct 30 01:09:53.009629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.971201    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02c7785-9248-485d-8d94-23290c4f6c0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:53.009629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:52.971345    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv7gp\" (UniqueName: \"kubernetes.io/projected/b02c7785-9248-485d-8d94-23290c4f6c0d-kube-api-access-sv7gp\") pod \"payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:53.054554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.054469    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02c7785-9248-485d-8d94-23290c4f6c0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:53.080754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.069619    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"]
Oct 30 01:09:53.298680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.298638    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv7gp\" (UniqueName: \"kubernetes.io/projected/b02c7785-9248-485d-8d94-23290c4f6c0d-kube-api-access-sv7gp\") pod \"payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:53.309714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.309677    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:53.349311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.349247    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:53.366703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.366605    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"]
Oct 30 01:09:53.457042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.456787    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b950c829-f560-490a-8265-f6ef547abfee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:53.457042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.456881    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rstb\" (UniqueName: \"kubernetes.io/projected/b950c829-f560-490a-8265-f6ef547abfee-kube-api-access-9rstb\") pod \"payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:53.558239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.558147    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b950c829-f560-490a-8265-f6ef547abfee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:53.558425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.558260    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rstb\" (UniqueName: \"kubernetes.io/projected/b950c829-f560-490a-8265-f6ef547abfee-kube-api-access-9rstb\") pod \"payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:53.558937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.558916    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b950c829-f560-490a-8265-f6ef547abfee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:53.577187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.577156    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rstb\" (UniqueName: \"kubernetes.io/projected/b950c829-f560-490a-8265-f6ef547abfee-kube-api-access-9rstb\") pod \"payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:53.670401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.670374    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"]
Oct 30 01:09:53.671757 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:09:53.671730    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02c7785_9248_485d_8d94_23290c4f6c0d.slice/crio-34e9ba98532904601a02cf55b21193a93de6befeca81912b27b083c528182e95 WatchSource:0}: Error finding container 34e9ba98532904601a02cf55b21193a93de6befeca81912b27b083c528182e95: Status 404 returned error can't find the container with id 34e9ba98532904601a02cf55b21193a93de6befeca81912b27b083c528182e95
Oct 30 01:09:53.708148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:53.708122    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:54.024306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.024278    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"]
Oct 30 01:09:54.025799 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:09:54.025764    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb950c829_f560_490a_8265_f6ef547abfee.slice/crio-3758b2c14d8400cf5cf7237a49866ec90fa0dc78a88adf41b7b019d37b58043b WatchSource:0}: Error finding container 3758b2c14d8400cf5cf7237a49866ec90fa0dc78a88adf41b7b019d37b58043b: Status 404 returned error can't find the container with id 3758b2c14d8400cf5cf7237a49866ec90fa0dc78a88adf41b7b019d37b58043b
Oct 30 01:09:54.366170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.366079    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" event={"ID":"b02c7785-9248-485d-8d94-23290c4f6c0d","Type":"ContainerStarted","Data":"e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f"}
Oct 30 01:09:54.366170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.366140    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" event={"ID":"b02c7785-9248-485d-8d94-23290c4f6c0d","Type":"ContainerStarted","Data":"34e9ba98532904601a02cf55b21193a93de6befeca81912b27b083c528182e95"}
Oct 30 01:09:54.366377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.366178    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:09:54.368154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.368129    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" event={"ID":"b950c829-f560-490a-8265-f6ef547abfee","Type":"ContainerStarted","Data":"a36f637b90fca3c6ac4585d4fbb892926915d19e915dfc07fb6591ce45392c85"}
Oct 30 01:09:54.368291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.368174    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" event={"ID":"b950c829-f560-490a-8265-f6ef547abfee","Type":"ContainerStarted","Data":"3758b2c14d8400cf5cf7237a49866ec90fa0dc78a88adf41b7b019d37b58043b"}
Oct 30 01:09:54.368291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.368200    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:09:54.410148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.410082    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" podStartSLOduration=2.410059366 podStartE2EDuration="2.410059366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:54.404045116 +0000 UTC m=+1635.951735631" watchObservedRunningTime="2026-10-30 01:09:54.410059366 +0000 UTC m=+1635.957749864"
Oct 30 01:09:54.442797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.442737    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" podStartSLOduration=2.442722619 podStartE2EDuration="2.442722619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:54.441255113 +0000 UTC m=+1635.988945609" watchObservedRunningTime="2026-10-30 01:09:54.442722619 +0000 UTC m=+1635.990413115"
Oct 30 01:09:54.759968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:54.759939    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:09:55.821231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:55.821197    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:09:59.009628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.009591    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"]
Oct 30 01:09:59.015515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.015489    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.048434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.047101    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"]
Oct 30 01:09:59.049435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.049410    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-64f472-openshift-default-dockercfg-vx8t4\""
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172227    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172327    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-data\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-credential-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172433    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d185da3d-11d2-401a-9c03-73f557d4d7f2-istiod-ca-cert\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172544    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-podinfo\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172616    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfjt\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-kube-api-access-pzfjt\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172673    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-token\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172703    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-certs\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.175625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.172763    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-envoy\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.276652    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.276767    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-data\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.276821    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-credential-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.276874    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d185da3d-11d2-401a-9c03-73f557d4d7f2-istiod-ca-cert\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.276912    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-podinfo\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.276980    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfjt\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-kube-api-access-pzfjt\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.277037    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-token\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.277068    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-certs\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.277529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.277147    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-envoy\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.296557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.282361    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d185da3d-11d2-401a-9c03-73f557d4d7f2-istiod-ca-cert\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.296825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.292319    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.296936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.292651    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-data\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.297107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.292890    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-credential-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.297238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.294072    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-certs\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.297674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.297651    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-envoy\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.336497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.334071    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-podinfo\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.337299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.337269    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-token\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.338604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.338581    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfjt\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-kube-api-access-pzfjt\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.637429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.637346    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:09:59.976576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.976543    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"]
Oct 30 01:09:59.977957 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:09:59.977929    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd185da3d_11d2_401a_9c03_73f557d4d7f2.slice/crio-7abb7fd4312c8e7199cbcf54ca43ce2f3a382310657e1309b02e8a9dcbe70f3a WatchSource:0}: Error finding container 7abb7fd4312c8e7199cbcf54ca43ce2f3a382310657e1309b02e8a9dcbe70f3a: Status 404 returned error can't find the container with id 7abb7fd4312c8e7199cbcf54ca43ce2f3a382310657e1309b02e8a9dcbe70f3a
Oct 30 01:09:59.981063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.981023    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:59.981188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.981111    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:09:59.981188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:09:59.981163    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:00.530443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:00.530401    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" event={"ID":"d185da3d-11d2-401a-9c03-73f557d4d7f2","Type":"ContainerStarted","Data":"fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1"}
Oct 30 01:10:00.530874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:00.530506    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" event={"ID":"d185da3d-11d2-401a-9c03-73f557d4d7f2","Type":"ContainerStarted","Data":"7abb7fd4312c8e7199cbcf54ca43ce2f3a382310657e1309b02e8a9dcbe70f3a"}
Oct 30 01:10:00.590477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:00.590402    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podStartSLOduration=2.590384697 podStartE2EDuration="2.590384697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:00.586391981 +0000 UTC m=+1642.134082477" watchObservedRunningTime="2026-10-30 01:10:00.590384697 +0000 UTC m=+1642.138075195"
Oct 30 01:10:00.637753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:00.637719    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:10:01.644941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:01.644907    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:10:02.551093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:02.551060    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:10:02.552217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:02.552197    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:10:04.995077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:04.995043    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"]
Oct 30 01:10:05.000390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.000370    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.018502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.018473    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"]
Oct 30 01:10:05.023702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.023668    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-64f472\""
Oct 30 01:10:05.023871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.023770    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-64f472-dockercfg-c7md2\""
Oct 30 01:10:05.388011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.387913    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905cca2-375e-46a5-9845-38ab8171ea14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.388409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.388383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rd2\" (UniqueName: \"kubernetes.io/projected/4905cca2-375e-46a5-9845-38ab8171ea14-kube-api-access-64rd2\") pod \"payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.407346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.407319    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:10:05.408797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.408579    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:10:05.481692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.481638    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"]
Oct 30 01:10:05.498024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.497987    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905cca2-375e-46a5-9845-38ab8171ea14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.498589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.498556    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rd2\" (UniqueName: \"kubernetes.io/projected/4905cca2-375e-46a5-9845-38ab8171ea14-kube-api-access-64rd2\") pod \"payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.499890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.499867    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.500754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.500303    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905cca2-375e-46a5-9845-38ab8171ea14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.510953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.510927    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"]
Oct 30 01:10:05.523937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.519784    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rd2\" (UniqueName: \"kubernetes.io/projected/4905cca2-375e-46a5-9845-38ab8171ea14-kube-api-access-64rd2\") pod \"payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.600259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.600225    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/f0b4d577-6ef2-427c-8bd9-527086752e20-kube-api-access-xz9bj\") pod \"payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.600448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.600418    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b4d577-6ef2-427c-8bd9-527086752e20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.617250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.617219    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:05.702047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.702009    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b4d577-6ef2-427c-8bd9-527086752e20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.702381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.702350    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/f0b4d577-6ef2-427c-8bd9-527086752e20-kube-api-access-xz9bj\") pod \"payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.703583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.703561    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b4d577-6ef2-427c-8bd9-527086752e20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.729177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.727619    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/f0b4d577-6ef2-427c-8bd9-527086752e20-kube-api-access-xz9bj\") pod \"payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.822697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.822663    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:05.948500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:05.948415    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"]
Oct 30 01:10:05.948653 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:10:05.948508    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4905cca2_375e_46a5_9845_38ab8171ea14.slice/crio-76b3569bb1594eec259acd00678ce5cbc1e9ee12afbace0703475e078bb66f56 WatchSource:0}: Error finding container 76b3569bb1594eec259acd00678ce5cbc1e9ee12afbace0703475e078bb66f56: Status 404 returned error can't find the container with id 76b3569bb1594eec259acd00678ce5cbc1e9ee12afbace0703475e078bb66f56
Oct 30 01:10:06.136342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.136314    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"]
Oct 30 01:10:06.137364 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:10:06.137327    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b4d577_6ef2_427c_8bd9_527086752e20.slice/crio-c272bf5c29d4084f2a4974ccf7d4acfd1680c5b202d9e89c8d75325325df20fb WatchSource:0}: Error finding container c272bf5c29d4084f2a4974ccf7d4acfd1680c5b202d9e89c8d75325325df20fb: Status 404 returned error can't find the container with id c272bf5c29d4084f2a4974ccf7d4acfd1680c5b202d9e89c8d75325325df20fb
Oct 30 01:10:06.598807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.598709    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" event={"ID":"f0b4d577-6ef2-427c-8bd9-527086752e20","Type":"ContainerStarted","Data":"2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56"}
Oct 30 01:10:06.598807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.598773    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" event={"ID":"f0b4d577-6ef2-427c-8bd9-527086752e20","Type":"ContainerStarted","Data":"c272bf5c29d4084f2a4974ccf7d4acfd1680c5b202d9e89c8d75325325df20fb"}
Oct 30 01:10:06.599055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.598812    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:06.600589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.600563    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" event={"ID":"4905cca2-375e-46a5-9845-38ab8171ea14","Type":"ContainerStarted","Data":"b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772"}
Oct 30 01:10:06.600729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.600609    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" event={"ID":"4905cca2-375e-46a5-9845-38ab8171ea14","Type":"ContainerStarted","Data":"76b3569bb1594eec259acd00678ce5cbc1e9ee12afbace0703475e078bb66f56"}
Oct 30 01:10:06.600729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.600660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:06.655855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.655807    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" podStartSLOduration=1.655793823 podStartE2EDuration="1.655793823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:06.649475717 +0000 UTC m=+1648.197166207" watchObservedRunningTime="2026-10-30 01:10:06.655793823 +0000 UTC m=+1648.203484318"
Oct 30 01:10:06.706985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:06.706935    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" podStartSLOduration=2.706921241 podStartE2EDuration="2.706921241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:06.704858335 +0000 UTC m=+1648.252548831" watchObservedRunningTime="2026-10-30 01:10:06.706921241 +0000 UTC m=+1648.254611737"
Oct 30 01:10:09.106008 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:10:09.105969    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:10:17.628523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:17.628492    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:10:17.634268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:17.634238    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:10:17.704634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:17.704599    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:10:19.663013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:19.662923    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"]
Oct 30 01:10:19.686568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:19.686530    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07410c-fcp9d"]
Oct 30 01:10:20.473212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:20.471619    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"]
Oct 30 01:10:20.517270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:20.517232    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-13a3e2-lldwp"]
Oct 30 01:10:20.654244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:20.654148    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a12606-b93f-4352-927a-2cdf6b5a70f5" path="/var/lib/kubelet/pods/a6a12606-b93f-4352-927a-2cdf6b5a70f5/volumes"
Oct 30 01:10:20.655307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:20.655265    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f697bb2a-fc15-477c-9130-9f93822e515a" path="/var/lib/kubelet/pods/f697bb2a-fc15-477c-9130-9f93822e515a/volumes"
Oct 30 01:10:21.458483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:21.393569    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:21.870540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:21.866648    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:21.900585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:21.900536    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:22.107327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.107275    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:10:22.145664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.145556    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:22.403735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.403622    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:22.404563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.404537    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:10:22.479482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.467805    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:22.479482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.468184    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:10:22.629496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.625952    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:22.649874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.649820    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:22.707491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.704790    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 01:10:22.870049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.869530    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:10:22.984534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:22.977774    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:23.190675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:23.190632    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:10:23.204551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:23.204520    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:10:23.204844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:23.204826    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:10:23.205102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:23.205067    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:10:23.205756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:23.205740    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:10:27.521937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:27.521898    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"]
Oct 30 01:10:27.522833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:27.522792    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy" containerID="cri-o://011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a" gracePeriod=30
Oct 30 01:10:28.913479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:28.906901    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"]
Oct 30 01:10:29.089800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.089761    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"]
Oct 30 01:10:29.133611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.090100    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.170958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.163004    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4dde48-openshift-default-dockercfg-6cb8d\""
Oct 30 01:10:29.327900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.327861    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-certs\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.329090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.329056    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-podinfo\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.329318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.329298    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-socket\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.329592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.329568    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-data\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.329770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.329749    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ck9f\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-kube-api-access-9ck9f\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.329993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.329975    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de5c9d-68d2-4468-be09-b9eee683918d-istiod-ca-cert\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.330148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.330133    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-envoy\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.330285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.330270    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-credential-socket\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.330498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.330434    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-token\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.559614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.559572    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-certs\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.559931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.559906    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-podinfo\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.560084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.560069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-socket\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.560351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.560331    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-data\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.560507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.560492    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ck9f\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-kube-api-access-9ck9f\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.560742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.560728    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de5c9d-68d2-4468-be09-b9eee683918d-istiod-ca-cert\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.560893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.560877    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-envoy\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.561042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.561027    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-credential-socket\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.561214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.561200    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-token\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.593480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.567054    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-data\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.619201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.567306    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-certs\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.619499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.594294    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de5c9d-68d2-4468-be09-b9eee683918d-istiod-ca-cert\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.626903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.626871    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-socket\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.641473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.641426    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-podinfo\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.717087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.717053    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-envoy\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.721283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.721249    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-credential-socket\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.723180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.723150    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-token\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.724400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.723860    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ck9f\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-kube-api-access-9ck9f\") pod \"e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") " pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.802314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.802227    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"]
Oct 30 01:10:29.810949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.810920    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:29.823751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.821606    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"]
Oct 30 01:10:29.826152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.825593    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:10:29.826152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.825751    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:10:29.826152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.826058    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:29.828141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.827245    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:10:29.828141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.827545    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:10:29.907524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.907114    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:29.907524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.907226    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxdt\" (UniqueName: \"kubernetes.io/projected/d642b1d9-3425-43e0-9e88-70009b08a4f8-kube-api-access-7rxdt\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:29.907524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:29.907295    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.009258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.009218    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.009756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.009273    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxdt\" (UniqueName: \"kubernetes.io/projected/d642b1d9-3425-43e0-9e88-70009b08a4f8-kube-api-access-7rxdt\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.009756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.009313    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.010137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.010106    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.011951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.011924    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.028521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.028484    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxdt\" (UniqueName: \"kubernetes.io/projected/d642b1d9-3425-43e0-9e88-70009b08a4f8-kube-api-access-7rxdt\") pod \"maas-api-revoke-keys-e2e-api-b-da60de-qs4wq\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.177522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.176369    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:30.236644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.236604    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"]
Oct 30 01:10:30.261340 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:10:30.261297    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8de5c9d_68d2_4468_be09_b9eee683918d.slice/crio-87f5ef63b02c64499457f71a4b0dbb98947f37aa3076e2c8b5642dc173fb26dc WatchSource:0}: Error finding container 87f5ef63b02c64499457f71a4b0dbb98947f37aa3076e2c8b5642dc173fb26dc: Status 404 returned error can't find the container with id 87f5ef63b02c64499457f71a4b0dbb98947f37aa3076e2c8b5642dc173fb26dc
Oct 30 01:10:30.274474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.273403    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:30.274474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.273530    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:30.274474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.273579    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:10:30.372937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.372909    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" event={"ID":"c8de5c9d-68d2-4468-be09-b9eee683918d","Type":"ContainerStarted","Data":"87f5ef63b02c64499457f71a4b0dbb98947f37aa3076e2c8b5642dc173fb26dc"}
Oct 30 01:10:30.561727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:30.561689    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"]
Oct 30 01:10:30.561901 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:10:30.561876    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd642b1d9_3425_43e0_9e88_70009b08a4f8.slice/crio-bcc0c832620f2bb36678b8c3b62fbcd684091e62e336c3c53dab35803c5cd3c6 WatchSource:0}: Error finding container bcc0c832620f2bb36678b8c3b62fbcd684091e62e336c3c53dab35803c5cd3c6: Status 404 returned error can't find the container with id bcc0c832620f2bb36678b8c3b62fbcd684091e62e336c3c53dab35803c5cd3c6
Oct 30 01:10:31.385704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:31.385669    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" event={"ID":"c8de5c9d-68d2-4468-be09-b9eee683918d","Type":"ContainerStarted","Data":"44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3"}
Oct 30 01:10:31.388193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:31.388155    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq" event={"ID":"d642b1d9-3425-43e0-9e88-70009b08a4f8","Type":"ContainerStarted","Data":"2c1372ae8b5f0483c328d6e1841592f4b0e8004e993a874a63733637bcf8e7fd"}
Oct 30 01:10:31.388363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:31.388220    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq" event={"ID":"d642b1d9-3425-43e0-9e88-70009b08a4f8","Type":"ContainerStarted","Data":"bcc0c832620f2bb36678b8c3b62fbcd684091e62e336c3c53dab35803c5cd3c6"}
Oct 30 01:10:31.440744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:31.440697    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podStartSLOduration=3.440682111 podStartE2EDuration="3.440682111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:31.435940626 +0000 UTC m=+1672.983631123" watchObservedRunningTime="2026-10-30 01:10:31.440682111 +0000 UTC m=+1672.988372606"
Oct 30 01:10:31.476337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:31.476283    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq" podStartSLOduration=2.476269462 podStartE2EDuration="2.476269462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:31.471440556 +0000 UTC m=+1673.019131052" watchObservedRunningTime="2026-10-30 01:10:31.476269462 +0000 UTC m=+1673.023959959"
Oct 30 01:10:31.826770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:31.826737    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:31.832743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:31.832718    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:32.482924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:32.480534    5104 generic.go:356] "Generic (PLEG): container finished" podID="d642b1d9-3425-43e0-9e88-70009b08a4f8" containerID="2c1372ae8b5f0483c328d6e1841592f4b0e8004e993a874a63733637bcf8e7fd" exitCode=0
Oct 30 01:10:32.482924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:32.481017    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq" event={"ID":"d642b1d9-3425-43e0-9e88-70009b08a4f8","Type":"ContainerDied","Data":"2c1372ae8b5f0483c328d6e1841592f4b0e8004e993a874a63733637bcf8e7fd"}
Oct 30 01:10:32.482924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:32.481357    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:32.485468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:32.485407    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:10:32.857813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:32.857786    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:10:33.018769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.018728    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-envoy\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.018960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.018859    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-socket\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.018960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.018916    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-data\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.018960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.018943    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-credential-socket\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.019220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.018980    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc10b078-6065-4c3b-9655-306808b43f55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc10b078-6065-4c3b-9655-306808b43f55-istio-podinfo\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.019220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019065    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-istio-token\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-istio-token\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.019220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019152    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-kube-api-access-fm48k\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-kube-api-access-fm48k\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.019220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019175    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-socket" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:33.019444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019314    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc10b078-6065-4c3b-9655-306808b43f55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc10b078-6065-4c3b-9655-306808b43f55-istiod-ca-cert\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.019444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019312    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-data" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:33.019444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019370    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-certs\") pod \"fc10b078-6065-4c3b-9655-306808b43f55\" (UID: \"fc10b078-6065-4c3b-9655-306808b43f55\") "
Oct 30 01:10:33.019444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019432    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-credential-socket" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:33.019899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019873    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-certs" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:33.020006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.019963    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc10b078-6065-4c3b-9655-306808b43f55-istiod-ca-cert" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:33.020297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.020272    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc10b078-6065-4c3b-9655-306808b43f55-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.020404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.020302    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.020404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.020318    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.020404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.020336    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.020404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.020394    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.022445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.022409    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-envoy" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:33.022557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.022507    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc10b078-6065-4c3b-9655-306808b43f55-istio-podinfo" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:33.022631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.022582    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-kube-api-access-fm48k" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "kube-api-access-fm48k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:33.022631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.022620    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-istio-token" pod "fc10b078-6065-4c3b-9655-306808b43f55" (UID: "fc10b078-6065-4c3b-9655-306808b43f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:33.122314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.122213    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm48k\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-kube-api-access-fm48k\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.122314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.122255    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc10b078-6065-4c3b-9655-306808b43f55-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.122314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.122270    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc10b078-6065-4c3b-9655-306808b43f55-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.122314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.122283    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc10b078-6065-4c3b-9655-306808b43f55-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.492852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.492818    5104 generic.go:356] "Generic (PLEG): container finished" podID="fc10b078-6065-4c3b-9655-306808b43f55" containerID="011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a" exitCode=0
Oct 30 01:10:33.493365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.492878    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"
Oct 30 01:10:33.493365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.492899    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" event={"ID":"fc10b078-6065-4c3b-9655-306808b43f55","Type":"ContainerDied","Data":"011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a"}
Oct 30 01:10:33.493365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.492989    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp" event={"ID":"fc10b078-6065-4c3b-9655-306808b43f55","Type":"ContainerDied","Data":"96e538ce8ab35e1269f39f922f4fd78f4adc126a44f6ec5075cefd7a509f0de8"}
Oct 30 01:10:33.493365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.493018    5104 scope.go:122] "RemoveContainer" containerID="011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a"
Oct 30 01:10:33.516881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.516858    5104 scope.go:122] "RemoveContainer" containerID="93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe"
Oct 30 01:10:33.530772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.530750    5104 scope.go:122] "RemoveContainer" containerID="011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a"
Oct 30 01:10:33.531142 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:10:33.531115    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a\": container with ID starting with 011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a not found: ID does not exist" containerID="011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a"
Oct 30 01:10:33.531262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.531147    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a"} err="failed to get container status \"011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a\": rpc error: code = NotFound desc = could not find container \"011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a\": container with ID starting with 011abb330b878b955780e6c71aa3759fc1e1d2cf3bd7d1dce1dbdd12895a710a not found: ID does not exist"
Oct 30 01:10:33.531262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.531166    5104 scope.go:122] "RemoveContainer" containerID="93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe"
Oct 30 01:10:33.531489 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:10:33.531472    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe\": container with ID starting with 93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe not found: ID does not exist" containerID="93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe"
Oct 30 01:10:33.531545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.531495    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe"} err="failed to get container status \"93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe\": rpc error: code = NotFound desc = could not find container \"93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe\": container with ID starting with 93398bdc09d990d028aaa4937e164ac8795668a9cbb978a0c0ba5d41e230d9fe not found: ID does not exist"
Oct 30 01:10:33.554484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.554430    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"]
Oct 30 01:10:33.562968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.562938    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-07410c-openshift-default-b6cc5f76f-fphzp"]
Oct 30 01:10:33.692381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.692350    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:33.834001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.833926    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d642b1d9-3425-43e0-9e88-70009b08a4f8-kube-api-access-7rxdt\" (UniqueName: \"kubernetes.io/projected/d642b1d9-3425-43e0-9e88-70009b08a4f8-kube-api-access-7rxdt\") pod \"d642b1d9-3425-43e0-9e88-70009b08a4f8\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") "
Oct 30 01:10:33.834325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.834274    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-openshift-ca\") pod \"d642b1d9-3425-43e0-9e88-70009b08a4f8\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") "
Oct 30 01:10:33.834325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.834322    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-xks-ca\") pod \"d642b1d9-3425-43e0-9e88-70009b08a4f8\" (UID: \"d642b1d9-3425-43e0-9e88-70009b08a4f8\") "
Oct 30 01:10:33.834874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.834848    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-openshift-ca" pod "d642b1d9-3425-43e0-9e88-70009b08a4f8" (UID: "d642b1d9-3425-43e0-9e88-70009b08a4f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:33.835235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.835177    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.837042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.837019    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-xks-ca" pod "d642b1d9-3425-43e0-9e88-70009b08a4f8" (UID: "d642b1d9-3425-43e0-9e88-70009b08a4f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:33.837159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.837103    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d642b1d9-3425-43e0-9e88-70009b08a4f8-kube-api-access-7rxdt" pod "d642b1d9-3425-43e0-9e88-70009b08a4f8" (UID: "d642b1d9-3425-43e0-9e88-70009b08a4f8"). InnerVolumeSpecName "kube-api-access-7rxdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:33.936754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.936715    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642b1d9-3425-43e0-9e88-70009b08a4f8-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:33.936754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:33.936751    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxdt\" (UniqueName: \"kubernetes.io/projected/d642b1d9-3425-43e0-9e88-70009b08a4f8-kube-api-access-7rxdt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.507027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:34.506987    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"
Oct 30 01:10:34.507520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:34.507059    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq" event={"ID":"d642b1d9-3425-43e0-9e88-70009b08a4f8","Type":"ContainerDied","Data":"bcc0c832620f2bb36678b8c3b62fbcd684091e62e336c3c53dab35803c5cd3c6"}
Oct 30 01:10:34.507520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:34.507123    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc0c832620f2bb36678b8c3b62fbcd684091e62e336c3c53dab35803c5cd3c6"
Oct 30 01:10:34.582645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:34.582606    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc10b078-6065-4c3b-9655-306808b43f55" path="/var/lib/kubelet/pods/fc10b078-6065-4c3b-9655-306808b43f55/volumes"
Oct 30 01:10:39.114772 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:10:39.114722    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:10:40.481825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:40.481780    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"]
Oct 30 01:10:44.522951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:44.522920    5104 scope.go:122] "RemoveContainer" containerID="e931f770e5bdab744c6acb1061e55c851a5b8691b5d595782882cb4569e33ee5"
Oct 30 01:10:44.534657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:44.534628    5104 scope.go:122] "RemoveContainer" containerID="fd042e985e18021d2cd7f14037efa41c4d04ed4c031023dd7fefeb84ff3691c3"
Oct 30 01:10:45.482640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.482593    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh" podUID="715e21c9-07e5-44e8-9daf-08b012575a61" containerName="payload-processing" containerID="cri-o://7eb10c87ad16fd2166960830726ae09c6150e64442a23f9ef1e8d415c89cc979" gracePeriod=25
Oct 30 01:10:45.742538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.739194    5104 generic.go:356] "Generic (PLEG): container finished" podID="715e21c9-07e5-44e8-9daf-08b012575a61" containerID="7eb10c87ad16fd2166960830726ae09c6150e64442a23f9ef1e8d415c89cc979" exitCode=0
Oct 30 01:10:45.742538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.739401    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh" event={"ID":"715e21c9-07e5-44e8-9daf-08b012575a61","Type":"ContainerDied","Data":"7eb10c87ad16fd2166960830726ae09c6150e64442a23f9ef1e8d415c89cc979"}
Oct 30 01:10:45.810362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.810329    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:10:45.892122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.892079    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715e21c9-07e5-44e8-9daf-08b012575a61-kube-api-access-gklnl\" (UniqueName: \"kubernetes.io/projected/715e21c9-07e5-44e8-9daf-08b012575a61-kube-api-access-gklnl\") pod \"715e21c9-07e5-44e8-9daf-08b012575a61\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") "
Oct 30 01:10:45.892548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.892520    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715e21c9-07e5-44e8-9daf-08b012575a61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e21c9-07e5-44e8-9daf-08b012575a61-plugins-config-volume\") pod \"715e21c9-07e5-44e8-9daf-08b012575a61\" (UID: \"715e21c9-07e5-44e8-9daf-08b012575a61\") "
Oct 30 01:10:45.892995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.892966    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715e21c9-07e5-44e8-9daf-08b012575a61-plugins-config-volume" pod "715e21c9-07e5-44e8-9daf-08b012575a61" (UID: "715e21c9-07e5-44e8-9daf-08b012575a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:45.893234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.893209    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e21c9-07e5-44e8-9daf-08b012575a61-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:45.895437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.895276    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715e21c9-07e5-44e8-9daf-08b012575a61-kube-api-access-gklnl" pod "715e21c9-07e5-44e8-9daf-08b012575a61" (UID: "715e21c9-07e5-44e8-9daf-08b012575a61"). InnerVolumeSpecName "kube-api-access-gklnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:45.995301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:45.995211    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gklnl\" (UniqueName: \"kubernetes.io/projected/715e21c9-07e5-44e8-9daf-08b012575a61-kube-api-access-gklnl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:10:46.749214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:46.749178    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"
Oct 30 01:10:46.749214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:46.749208    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh" event={"ID":"715e21c9-07e5-44e8-9daf-08b012575a61","Type":"ContainerDied","Data":"68f83acdb344e3ee567f5c4e92ccb4775e675beec40245d71e628d3b44d20f9f"}
Oct 30 01:10:46.749704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:46.749267    5104 scope.go:122] "RemoveContainer" containerID="7eb10c87ad16fd2166960830726ae09c6150e64442a23f9ef1e8d415c89cc979"
Oct 30 01:10:46.799150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:46.799118    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"]
Oct 30 01:10:46.810429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:46.810392    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da60de-877f4976-zv2gh"]
Oct 30 01:10:48.586284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:10:48.586244    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715e21c9-07e5-44e8-9daf-08b012575a61" path="/var/lib/kubelet/pods/715e21c9-07e5-44e8-9daf-08b012575a61/volumes"
Oct 30 01:11:02.387346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.387304    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"]
Oct 30 01:11:02.388758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.388731    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy"
Oct 30 01:11:02.388758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.388758    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy"
Oct 30 01:11:02.388971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.388834    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d642b1d9-3425-43e0-9e88-70009b08a4f8" containerName="revoke-keys"
Oct 30 01:11:02.388971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.388844    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d642b1d9-3425-43e0-9e88-70009b08a4f8" containerName="revoke-keys"
Oct 30 01:11:02.388971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.388867    5104 cpu_manager.go:414] "containerMap: removing container" podUID="715e21c9-07e5-44e8-9daf-08b012575a61" containerName="payload-processing"
Oct 30 01:11:02.388971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.388874    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715e21c9-07e5-44e8-9daf-08b012575a61" containerName="payload-processing"
Oct 30 01:11:02.389187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.389092    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d642b1d9-3425-43e0-9e88-70009b08a4f8" containerName="revoke-keys"
Oct 30 01:11:02.389187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.389112    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="715e21c9-07e5-44e8-9daf-08b012575a61" containerName="payload-processing"
Oct 30 01:11:02.389187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.389122    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy"
Oct 30 01:11:02.400757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.395626    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.423276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.423246    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-687471-openshift-default-dockercfg-nwb7x\""
Oct 30 01:11:02.439946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.439916    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"]
Oct 30 01:11:02.462246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462206    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-credential-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.462246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462249    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69c9213-1688-410c-8e09-06ac2dc53702-istio-podinfo\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.462539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462360    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-envoy\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.462539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462505    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-certs\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.462657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462561    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4482\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-kube-api-access-l4482\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.462699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462675    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69c9213-1688-410c-8e09-06ac2dc53702-istiod-ca-cert\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.462803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462782    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.462898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462881    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-istio-token\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.463002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.462986    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-data\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.571608    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-data\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.571773    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-credential-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.571804    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69c9213-1688-410c-8e09-06ac2dc53702-istio-podinfo\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.571867    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-envoy\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.571968    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-certs\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.572022    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4482\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-kube-api-access-l4482\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.572078    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69c9213-1688-410c-8e09-06ac2dc53702-istiod-ca-cert\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.572189    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.576491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.572304    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-istio-token\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.592477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.582259    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-data\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.592477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.582694    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-certs\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.592477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.582982    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-credential-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.592477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.589508    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69c9213-1688-410c-8e09-06ac2dc53702-istio-podinfo\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.592477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.591475    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69c9213-1688-410c-8e09-06ac2dc53702-istiod-ca-cert\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.592477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.591757    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.619505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.610584    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-istio-token\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.620026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.619991    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-envoy\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.643538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.643399    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4482\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-kube-api-access-l4482\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:02.748514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:02.746382    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:03.030250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:03.030217    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"]
Oct 30 01:11:03.037532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:03.037493    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a087f059-dsbwm"]
Oct 30 01:11:03.275856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:03.275822    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"]
Oct 30 01:11:03.276613 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:03.276586    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69c9213_1688_410c_8e09_06ac2dc53702.slice/crio-8714dab35b714e6774ff8ac03f7420fe180f9741ca69e3193c3e06ae445fee5c WatchSource:0}: Error finding container 8714dab35b714e6774ff8ac03f7420fe180f9741ca69e3193c3e06ae445fee5c: Status 404 returned error can't find the container with id 8714dab35b714e6774ff8ac03f7420fe180f9741ca69e3193c3e06ae445fee5c
Oct 30 01:11:03.279332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:03.279294    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:03.279407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:03.279370    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:03.279407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:03.279400    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:04.080576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:04.080537    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" event={"ID":"a69c9213-1688-410c-8e09-06ac2dc53702","Type":"ContainerStarted","Data":"0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2"}
Oct 30 01:11:04.083197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:04.083166    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" event={"ID":"a69c9213-1688-410c-8e09-06ac2dc53702","Type":"ContainerStarted","Data":"8714dab35b714e6774ff8ac03f7420fe180f9741ca69e3193c3e06ae445fee5c"}
Oct 30 01:11:04.202480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:04.187151    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podStartSLOduration=2.187128932 podStartE2EDuration="2.187128932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:04.161895712 +0000 UTC m=+1705.709586208" watchObservedRunningTime="2026-10-30 01:11:04.187128932 +0000 UTC m=+1705.734819430"
Oct 30 01:11:04.651487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:04.649249    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3" path="/var/lib/kubelet/pods/bc8a1c63-e9f7-4358-83f0-eabccdd5b5e3/volumes"
Oct 30 01:11:04.751519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:04.751439    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:04.768562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:04.768241    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:05.090232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:05.090203    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:05.091338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:05.091315    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:11:08.310075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.310036    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"]
Oct 30 01:11:08.311114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.311090    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy"
Oct 30 01:11:08.311114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.311114    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy"
Oct 30 01:11:08.311405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.311389    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc10b078-6065-4c3b-9655-306808b43f55" containerName="istio-proxy"
Oct 30 01:11:08.317155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.317125    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.340437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.340412    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-56d1d9-openshift-default-dockercfg-hxhn9\""
Oct 30 01:11:08.348052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.348018    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"]
Oct 30 01:11:08.363089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363058    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-certs\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363239    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-podinfo\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363299    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-envoy\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363360    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-token\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363546    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-socket\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363638    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wpp2\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-kube-api-access-9wpp2\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363709    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-data\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363815    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-credential-socket\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.363958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.363855    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istiod-ca-cert\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.512718    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-podinfo\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.512786    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-envoy\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.512816    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-token\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.512986    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-socket\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.513066    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wpp2\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-kube-api-access-9wpp2\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.513163    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-data\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.513207    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-credential-socket\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.513231    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istiod-ca-cert\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.513284    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-certs\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.514479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.513828    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-certs\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.534513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.518059    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-envoy\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.534513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.518642    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-socket\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.534513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.518884    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-credential-socket\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.534513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.519109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-data\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.534513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.519776    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istiod-ca-cert\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.570571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.561679    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-token\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.596479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.594586    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-podinfo\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.665479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.662258    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wpp2\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-kube-api-access-9wpp2\") pod \"e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") " pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:08.966576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:08.965393    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:09.147304 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:11:09.147251    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:11:09.318613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.318586    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"]
Oct 30 01:11:09.320793 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:09.320766    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5d5e2e_e9b2_4ff2_957a_f736ea96503a.slice/crio-61ec9ce98a7bd57a0745dca8fb2000a17da6e8e59cea5fe799ba1320c3e30d4e WatchSource:0}: Error finding container 61ec9ce98a7bd57a0745dca8fb2000a17da6e8e59cea5fe799ba1320c3e30d4e: Status 404 returned error can't find the container with id 61ec9ce98a7bd57a0745dca8fb2000a17da6e8e59cea5fe799ba1320c3e30d4e
Oct 30 01:11:09.324479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.324429    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:09.324561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.324512    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:09.324561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.324544    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:09.564670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.564634    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"]
Oct 30 01:11:09.570075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.569996    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.585956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.585796    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:09.585956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.585916    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:09.586223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.585976    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:09.592780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.592750    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"]
Oct 30 01:11:09.593771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.593745    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:11:09.691247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.691202    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.691447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.691256    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.691447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.691436    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k252\" (UniqueName: \"kubernetes.io/projected/1efc5707-f19e-48d8-94e4-c2c8962685fc-kube-api-access-2k252\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.874529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.862594    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.874529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.862652    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.874529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.862834    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k252\" (UniqueName: \"kubernetes.io/projected/1efc5707-f19e-48d8-94e4-c2c8962685fc-kube-api-access-2k252\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.874529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.864042    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:09.919487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.905546    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"]
Oct 30 01:11:09.939492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.939440    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:09.992490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:09.992441    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-687471\""
Oct 30 01:11:10.012554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.012516    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"]
Oct 30 01:11:10.062112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.062075    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-687471-dockercfg-26g74\""
Oct 30 01:11:10.115357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.115301    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:10.192603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.143563    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:10.193770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.193734    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694rj\" (UniqueName: \"kubernetes.io/projected/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-kube-api-access-694rj\") pod \"payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:10.212503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.199237    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k252\" (UniqueName: \"kubernetes.io/projected/1efc5707-f19e-48d8-94e4-c2c8962685fc-kube-api-access-2k252\") pod \"maas-api-revoke-sub-f83ee24d-cvmf9\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:10.253596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.239014    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" event={"ID":"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a","Type":"ContainerStarted","Data":"f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767"}
Oct 30 01:11:10.253596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.239074    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" event={"ID":"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a","Type":"ContainerStarted","Data":"61ec9ce98a7bd57a0745dca8fb2000a17da6e8e59cea5fe799ba1320c3e30d4e"}
Oct 30 01:11:10.306479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.305741    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:10.320478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.312193    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:10.320478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.312255    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694rj\" (UniqueName: \"kubernetes.io/projected/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-kube-api-access-694rj\") pod \"payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:10.320478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.316009    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podStartSLOduration=2.315988029 podStartE2EDuration="2.315988029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:10.302143521 +0000 UTC m=+1711.849834021" watchObservedRunningTime="2026-10-30 01:11:10.315988029 +0000 UTC m=+1711.863678529"
Oct 30 01:11:10.332715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.332626    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-694rj\" (UniqueName: \"kubernetes.io/projected/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-kube-api-access-694rj\") pod \"payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:10.497134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.496789    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:10.556493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.556447    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:10.966423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.966382    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:10.972886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:10.972852    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:11.062621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.062588    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"]
Oct 30 01:11:11.064050 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:11.064021    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efc5707_f19e_48d8_94e4_c2c8962685fc.slice/crio-50c11346b115df27a399eb50f44f2e750a68ef08f54ad7eac6ec348458daa9d0 WatchSource:0}: Error finding container 50c11346b115df27a399eb50f44f2e750a68ef08f54ad7eac6ec348458daa9d0: Status 404 returned error can't find the container with id 50c11346b115df27a399eb50f44f2e750a68ef08f54ad7eac6ec348458daa9d0
Oct 30 01:11:11.083986 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:11.083619    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7803c74d_04f3_46b2_b0e8_552e5ad89b6c.slice/crio-9bb473bdf22e5d5dae5ba9dbb42f25f8e93c5695bc5604e35f4aa4499b9c8aa7 WatchSource:0}: Error finding container 9bb473bdf22e5d5dae5ba9dbb42f25f8e93c5695bc5604e35f4aa4499b9c8aa7: Status 404 returned error can't find the container with id 9bb473bdf22e5d5dae5ba9dbb42f25f8e93c5695bc5604e35f4aa4499b9c8aa7
Oct 30 01:11:11.089771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.089742    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"]
Oct 30 01:11:11.251255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.251216    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9" event={"ID":"1efc5707-f19e-48d8-94e4-c2c8962685fc","Type":"ContainerStarted","Data":"50c11346b115df27a399eb50f44f2e750a68ef08f54ad7eac6ec348458daa9d0"}
Oct 30 01:11:11.253213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.253186    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568" event={"ID":"7803c74d-04f3-46b2-b0e8-552e5ad89b6c","Type":"ContainerStarted","Data":"7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81"}
Oct 30 01:11:11.253405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.253239    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568" event={"ID":"7803c74d-04f3-46b2-b0e8-552e5ad89b6c","Type":"ContainerStarted","Data":"9bb473bdf22e5d5dae5ba9dbb42f25f8e93c5695bc5604e35f4aa4499b9c8aa7"}
Oct 30 01:11:11.253405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.253321    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:11.253539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.253513    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:11.254519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.254496    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:11:11.321540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:11.300416    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568" podStartSLOduration=2.300396339 podStartE2EDuration="2.300396339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:11.29478154 +0000 UTC m=+1712.842472037" watchObservedRunningTime="2026-10-30 01:11:11.300396339 +0000 UTC m=+1712.848086836"
Oct 30 01:11:12.269232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.269191    5104 generic.go:356] "Generic (PLEG): container finished" podID="1efc5707-f19e-48d8-94e4-c2c8962685fc" containerID="291b8e610ce234f0e0c423457378971133062ae068d8b2e306665baa97d1228f" exitCode=0
Oct 30 01:11:12.269409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.269268    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9" event={"ID":"1efc5707-f19e-48d8-94e4-c2c8962685fc","Type":"ContainerDied","Data":"291b8e610ce234f0e0c423457378971133062ae068d8b2e306665baa97d1228f"}
Oct 30 01:11:12.497621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.497575    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"]
Oct 30 01:11:12.504226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.504142    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:12.528105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.527845    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82e9bf-dockercfg-wpvhg\""
Oct 30 01:11:12.533468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.533430    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"]
Oct 30 01:11:12.566537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.563901    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82e9bf\""
Oct 30 01:11:12.823544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.821293    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:12.823544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.821492    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjknf\" (UniqueName: \"kubernetes.io/projected/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-kube-api-access-tjknf\") pod \"payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:12.946512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.931303    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:12.946512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.931472    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjknf\" (UniqueName: \"kubernetes.io/projected/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-kube-api-access-tjknf\") pod \"payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:12.946512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.933012    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:12.992585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:12.992339    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjknf\" (UniqueName: \"kubernetes.io/projected/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-kube-api-access-tjknf\") pod \"payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:13.125984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.125889    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:13.506517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.506491    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:13.583662 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:13.583631    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51768a8_3a1a_47be_8ad9_27eca95f9b9e.slice/crio-d4d5f8cfddb1abf3060247ab3a8ca943fe33ef81ab27d85a90cc38d8c86e2c87 WatchSource:0}: Error finding container d4d5f8cfddb1abf3060247ab3a8ca943fe33ef81ab27d85a90cc38d8c86e2c87: Status 404 returned error can't find the container with id d4d5f8cfddb1abf3060247ab3a8ca943fe33ef81ab27d85a90cc38d8c86e2c87
Oct 30 01:11:13.584842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.584714    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"]
Oct 30 01:11:13.676217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.676182    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-xks-ca\") pod \"1efc5707-f19e-48d8-94e4-c2c8962685fc\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") "
Oct 30 01:11:13.676390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.676244    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efc5707-f19e-48d8-94e4-c2c8962685fc-kube-api-access-2k252\" (UniqueName: \"kubernetes.io/projected/1efc5707-f19e-48d8-94e4-c2c8962685fc-kube-api-access-2k252\") pod \"1efc5707-f19e-48d8-94e4-c2c8962685fc\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") "
Oct 30 01:11:13.676390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.676284    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-openshift-ca\") pod \"1efc5707-f19e-48d8-94e4-c2c8962685fc\" (UID: \"1efc5707-f19e-48d8-94e4-c2c8962685fc\") "
Oct 30 01:11:13.676995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.676966    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-openshift-ca" pod "1efc5707-f19e-48d8-94e4-c2c8962685fc" (UID: "1efc5707-f19e-48d8-94e4-c2c8962685fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:13.677175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.677156    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:13.679043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.679018    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-xks-ca" pod "1efc5707-f19e-48d8-94e4-c2c8962685fc" (UID: "1efc5707-f19e-48d8-94e4-c2c8962685fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:13.679132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.679054    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efc5707-f19e-48d8-94e4-c2c8962685fc-kube-api-access-2k252" pod "1efc5707-f19e-48d8-94e4-c2c8962685fc" (UID: "1efc5707-f19e-48d8-94e4-c2c8962685fc"). InnerVolumeSpecName "kube-api-access-2k252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:13.778290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.778253    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efc5707-f19e-48d8-94e4-c2c8962685fc-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:13.778290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:13.778290    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k252\" (UniqueName: \"kubernetes.io/projected/1efc5707-f19e-48d8-94e4-c2c8962685fc-kube-api-access-2k252\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:14.310526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.302959    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"
Oct 30 01:11:14.310526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.304757    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9" event={"ID":"1efc5707-f19e-48d8-94e4-c2c8962685fc","Type":"ContainerDied","Data":"50c11346b115df27a399eb50f44f2e750a68ef08f54ad7eac6ec348458daa9d0"}
Oct 30 01:11:14.310526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.304827    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c11346b115df27a399eb50f44f2e750a68ef08f54ad7eac6ec348458daa9d0"
Oct 30 01:11:14.329518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.329066    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" event={"ID":"d51768a8-3a1a-47be-8ad9-27eca95f9b9e","Type":"ContainerStarted","Data":"ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae"}
Oct 30 01:11:14.329518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.329139    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" event={"ID":"d51768a8-3a1a-47be-8ad9-27eca95f9b9e","Type":"ContainerStarted","Data":"d4d5f8cfddb1abf3060247ab3a8ca943fe33ef81ab27d85a90cc38d8c86e2c87"}
Oct 30 01:11:14.333479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.331542    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:14.402277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.402213    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" podStartSLOduration=2.402189912 podStartE2EDuration="2.402189912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:14.385376806 +0000 UTC m=+1715.933067315" watchObservedRunningTime="2026-10-30 01:11:14.402189912 +0000 UTC m=+1715.949880410"
Oct 30 01:11:14.965592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:14.965252    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"]
Oct 30 01:11:15.164061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.164019    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1efc5707-f19e-48d8-94e4-c2c8962685fc" containerName="revoke-keys"
Oct 30 01:11:15.164061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.164059    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efc5707-f19e-48d8-94e4-c2c8962685fc" containerName="revoke-keys"
Oct 30 01:11:15.176352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.176314    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efc5707-f19e-48d8-94e4-c2c8962685fc" containerName="revoke-keys"
Oct 30 01:11:15.352681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.352594    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"]
Oct 30 01:11:15.352850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.352693    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.366368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.366340    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-56d1d9\""
Oct 30 01:11:15.366594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.366383    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-56d1d9-dockercfg-f6lgf\""
Oct 30 01:11:15.429559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.429525    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7wl\" (UniqueName: \"kubernetes.io/projected/88842834-7005-4c5d-bade-eddca63b948b-kube-api-access-4x7wl\") pod \"payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.429803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.429790    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88842834-7005-4c5d-bade-eddca63b948b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.531269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.531229    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88842834-7005-4c5d-bade-eddca63b948b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.531478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.531352    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7wl\" (UniqueName: \"kubernetes.io/projected/88842834-7005-4c5d-bade-eddca63b948b-kube-api-access-4x7wl\") pod \"payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.532005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.531978    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88842834-7005-4c5d-bade-eddca63b948b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.550419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.550389    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7wl\" (UniqueName: \"kubernetes.io/projected/88842834-7005-4c5d-bade-eddca63b948b-kube-api-access-4x7wl\") pod \"payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.669099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.669068    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:15.981940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:15.980513    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"]
Oct 30 01:11:16.456686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:16.456601    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" event={"ID":"88842834-7005-4c5d-bade-eddca63b948b","Type":"ContainerStarted","Data":"9da08a6905bff1aa05eded31b1c95d871660e522b9527575dd0390fb522428db"}
Oct 30 01:11:16.457111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:16.457080    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" event={"ID":"88842834-7005-4c5d-bade-eddca63b948b","Type":"ContainerStarted","Data":"7a5d2d229e9935ef80a3b8011bea21223cb7b11efe097c084b157233e7675218"}
Oct 30 01:11:16.457280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:16.457266    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:16.495509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:16.495425    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" podStartSLOduration=2.495403502 podStartE2EDuration="2.495403502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:16.49207265 +0000 UTC m=+1718.039763149" watchObservedRunningTime="2026-10-30 01:11:16.495403502 +0000 UTC m=+1718.043094000"
Oct 30 01:11:21.254483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.245917    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:11:21.305478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.304527    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"]
Oct 30 01:11:21.512773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.512692    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"]
Oct 30 01:11:21.523744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.513048    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.544485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.544439    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:21.549702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.549674    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:11:21.563441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.563412    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:21.563626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.563527    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:21.582096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.582056    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjcf\" (UniqueName: \"kubernetes.io/projected/db452e21-afc3-4265-8ac9-6decf011fbdc-kube-api-access-svjcf\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.582323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.582272    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.582521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.582408    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.684590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.684551    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.684779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.684751    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjcf\" (UniqueName: \"kubernetes.io/projected/db452e21-afc3-4265-8ac9-6decf011fbdc-kube-api-access-svjcf\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.684969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.684951    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.685300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.685276    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.687432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.687405    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.709420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.709386    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjcf\" (UniqueName: \"kubernetes.io/projected/db452e21-afc3-4265-8ac9-6decf011fbdc-kube-api-access-svjcf\") pod \"maas-api-revoke-sub-a1c0ed29-zcjrg\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:21.846103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:21.846018    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:22.160603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:22.160570    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"]
Oct 30 01:11:22.161952 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:22.161924    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb452e21_afc3_4265_8ac9_6decf011fbdc.slice/crio-a72a2e0c28500669d4b2413c8b8d0ff48045b2516c888d7cb6f66b0bc13cfdf7 WatchSource:0}: Error finding container a72a2e0c28500669d4b2413c8b8d0ff48045b2516c888d7cb6f66b0bc13cfdf7: Status 404 returned error can't find the container with id a72a2e0c28500669d4b2413c8b8d0ff48045b2516c888d7cb6f66b0bc13cfdf7
Oct 30 01:11:22.274239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:22.274206    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:11:22.765181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:22.765137    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg" event={"ID":"db452e21-afc3-4265-8ac9-6decf011fbdc","Type":"ContainerStarted","Data":"a72a2e0c28500669d4b2413c8b8d0ff48045b2516c888d7cb6f66b0bc13cfdf7"}
Oct 30 01:11:23.780266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:23.780221    5104 generic.go:356] "Generic (PLEG): container finished" podID="db452e21-afc3-4265-8ac9-6decf011fbdc" containerID="0364c4c248f069dc80326066bf02387d52e89e14334fc6cfaff8b4cef9243c3c" exitCode=0
Oct 30 01:11:23.780740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:23.780307    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg" event={"ID":"db452e21-afc3-4265-8ac9-6decf011fbdc","Type":"ContainerDied","Data":"0364c4c248f069dc80326066bf02387d52e89e14334fc6cfaff8b4cef9243c3c"}
Oct 30 01:11:25.516744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.516355    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:25.561561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.561516    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db452e21-afc3-4265-8ac9-6decf011fbdc-kube-api-access-svjcf\" (UniqueName: \"kubernetes.io/projected/db452e21-afc3-4265-8ac9-6decf011fbdc-kube-api-access-svjcf\") pod \"db452e21-afc3-4265-8ac9-6decf011fbdc\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") "
Oct 30 01:11:25.561785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.561642    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-xks-ca\") pod \"db452e21-afc3-4265-8ac9-6decf011fbdc\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") "
Oct 30 01:11:25.561785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.561705    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-openshift-ca\") pod \"db452e21-afc3-4265-8ac9-6decf011fbdc\" (UID: \"db452e21-afc3-4265-8ac9-6decf011fbdc\") "
Oct 30 01:11:25.562882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.562841    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-openshift-ca" pod "db452e21-afc3-4265-8ac9-6decf011fbdc" (UID: "db452e21-afc3-4265-8ac9-6decf011fbdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:25.564802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.564776    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db452e21-afc3-4265-8ac9-6decf011fbdc-kube-api-access-svjcf" pod "db452e21-afc3-4265-8ac9-6decf011fbdc" (UID: "db452e21-afc3-4265-8ac9-6decf011fbdc"). InnerVolumeSpecName "kube-api-access-svjcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:25.564911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.564879    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-xks-ca" pod "db452e21-afc3-4265-8ac9-6decf011fbdc" (UID: "db452e21-afc3-4265-8ac9-6decf011fbdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:25.664177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.664101    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svjcf\" (UniqueName: \"kubernetes.io/projected/db452e21-afc3-4265-8ac9-6decf011fbdc-kube-api-access-svjcf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.664177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.664129    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.664177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.664142    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db452e21-afc3-4265-8ac9-6decf011fbdc-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.822248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.822213    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg" event={"ID":"db452e21-afc3-4265-8ac9-6decf011fbdc","Type":"ContainerDied","Data":"a72a2e0c28500669d4b2413c8b8d0ff48045b2516c888d7cb6f66b0bc13cfdf7"}
Oct 30 01:11:25.822248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.822223    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"
Oct 30 01:11:25.822487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:25.822262    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a72a2e0c28500669d4b2413c8b8d0ff48045b2516c888d7cb6f66b0bc13cfdf7"
Oct 30 01:11:26.458738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.458709    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:11:26.712899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.712809    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"]
Oct 30 01:11:26.714229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.714205    5104 cpu_manager.go:414] "containerMap: removing container" podUID="db452e21-afc3-4265-8ac9-6decf011fbdc" containerName="revoke-keys"
Oct 30 01:11:26.714299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.714232    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db452e21-afc3-4265-8ac9-6decf011fbdc" containerName="revoke-keys"
Oct 30 01:11:26.714561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.714542    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="db452e21-afc3-4265-8ac9-6decf011fbdc" containerName="revoke-keys"
Oct 30 01:11:26.719763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.719742    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:26.721029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.721005    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"]
Oct 30 01:11:26.734226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.734199    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-29145c\""
Oct 30 01:11:26.790825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.790793    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-29145c-dockercfg-5r9cj\""
Oct 30 01:11:26.815709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.815664    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:26.816536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.816498    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nml\" (UniqueName: \"kubernetes.io/projected/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-kube-api-access-l9nml\") pod \"payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:26.989532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.973762    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:26.989532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.973973    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nml\" (UniqueName: \"kubernetes.io/projected/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-kube-api-access-l9nml\") pod \"payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:27.005558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:26.990660    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:27.428535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:27.353637    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nml\" (UniqueName: \"kubernetes.io/projected/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-kube-api-access-l9nml\") pod \"payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:27.527479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:27.525107    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:27.914185 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:27.914148    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b4e2db_b55a_428a_ae08_44d6378ccbf7.slice/crio-a377c6997b67103df5453c4a8edf68d6ffb15e4e9b33d569975fd458e4c7cfcb WatchSource:0}: Error finding container a377c6997b67103df5453c4a8edf68d6ffb15e4e9b33d569975fd458e4c7cfcb: Status 404 returned error can't find the container with id a377c6997b67103df5453c4a8edf68d6ffb15e4e9b33d569975fd458e4c7cfcb
Oct 30 01:11:27.917088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:27.917048    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"]
Oct 30 01:11:28.072385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:28.072343    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" event={"ID":"c6b4e2db-b55a-428a-ae08-44d6378ccbf7","Type":"ContainerStarted","Data":"a377c6997b67103df5453c4a8edf68d6ffb15e4e9b33d569975fd458e4c7cfcb"}
Oct 30 01:11:28.561032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:28.561004    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:11:29.101482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:29.101303    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" event={"ID":"c6b4e2db-b55a-428a-ae08-44d6378ccbf7","Type":"ContainerStarted","Data":"dd393ab5ad3833c43e68ca0acd607b8c40768b391c464436cdde00e64441b09e"}
Oct 30 01:11:29.123479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:29.119900    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:29.144947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:29.144896    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" podStartSLOduration=3.14488182 podStartE2EDuration="3.14488182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:29.14102244 +0000 UTC m=+1730.688712936" watchObservedRunningTime="2026-10-30 01:11:29.14488182 +0000 UTC m=+1730.692572315"
Oct 30 01:11:33.593629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:33.593580    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:11:33.594286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:33.593638    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:11:35.234739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:35.234707    5104 generic.go:356] "Generic (PLEG): container finished" podID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerID="fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1" exitCode=0
Oct 30 01:11:35.235207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:35.234765    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" event={"ID":"d185da3d-11d2-401a-9c03-73f557d4d7f2","Type":"ContainerDied","Data":"fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1"}
Oct 30 01:11:35.235269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:35.235247    5104 scope.go:122] "RemoveContainer" containerID="fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1"
Oct 30 01:11:35.238587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:35.238529    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:35.238768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:35.238734    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:35.238888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:35.238862    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:36.254118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:36.254079    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" event={"ID":"d185da3d-11d2-401a-9c03-73f557d4d7f2","Type":"ContainerStarted","Data":"3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172"}
Oct 30 01:11:36.638423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:36.638327    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:11:36.638677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:36.638654    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:36.638779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:36.638708    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:37.148526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.146089    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"]
Oct 30 01:11:37.174501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.170911    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.207123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.205817    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:37.207123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.205976    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:11:37.207123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.206044    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:37.207123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.206776    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:37.357725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.357680    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.358239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.357754    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2b6k\" (UniqueName: \"kubernetes.io/projected/88be93e3-58b9-4416-87f9-daefc7b22ac0-kube-api-access-w2b6k\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.358239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.358162    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.371571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.371521    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"]
Oct 30 01:11:37.503495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.498904    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2b6k\" (UniqueName: \"kubernetes.io/projected/88be93e3-58b9-4416-87f9-daefc7b22ac0-kube-api-access-w2b6k\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.503495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.499906    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.503495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.500005    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.528481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.509415    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.615663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.613830    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2b6k\" (UniqueName: \"kubernetes.io/projected/88be93e3-58b9-4416-87f9-daefc7b22ac0-kube-api-access-w2b6k\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.619955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.619925    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:37.639667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.639628    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:37.639871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.639690    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:37.750525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:37.750491    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:38.035858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:38.035764    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"]
Oct 30 01:11:38.044053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:38.044013    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c41a222-687cz"]
Oct 30 01:11:38.113023 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:11:38.112077    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88be93e3_58b9_4416_87f9_daefc7b22ac0.slice/crio-5ff8729c9c27d3eff2035be4940096480cd62d8acf54ac8a1c1e8f4caee0bc89 WatchSource:0}: Error finding container 5ff8729c9c27d3eff2035be4940096480cd62d8acf54ac8a1c1e8f4caee0bc89: Status 404 returned error can't find the container with id 5ff8729c9c27d3eff2035be4940096480cd62d8acf54ac8a1c1e8f4caee0bc89
Oct 30 01:11:38.114549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:38.114521    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"]
Oct 30 01:11:38.370432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:38.370389    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x" event={"ID":"88be93e3-58b9-4416-87f9-daefc7b22ac0","Type":"ContainerStarted","Data":"5ff8729c9c27d3eff2035be4940096480cd62d8acf54ac8a1c1e8f4caee0bc89"}
Oct 30 01:11:38.615567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:38.615110    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5010c401-fa9d-4254-bccb-2b52cd111014" path="/var/lib/kubelet/pods/5010c401-fa9d-4254-bccb-2b52cd111014/volumes"
Oct 30 01:11:38.638516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:38.638484    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:38.638684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:38.638536    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:39.140619 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:11:39.140580    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:11:39.436425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:39.436384    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x" event={"ID":"88be93e3-58b9-4416-87f9-daefc7b22ac0","Type":"ContainerStarted","Data":"1c66316f3b436e4a41d8c6007e2017cb22a20a194817e7800f3c2bd6074952b8"}
Oct 30 01:11:39.500479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:39.478119    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x" podStartSLOduration=2.478100911 podStartE2EDuration="2.478100911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:39.476010513 +0000 UTC m=+1741.023701014" watchObservedRunningTime="2026-10-30 01:11:39.478100911 +0000 UTC m=+1741.025791409"
Oct 30 01:11:39.546844 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:11:39.546808    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88be93e3_58b9_4416_87f9_daefc7b22ac0.slice/crio-1c66316f3b436e4a41d8c6007e2017cb22a20a194817e7800f3c2bd6074952b8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88be93e3_58b9_4416_87f9_daefc7b22ac0.slice/crio-conmon-1c66316f3b436e4a41d8c6007e2017cb22a20a194817e7800f3c2bd6074952b8.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:11:39.638473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:39.638275    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:39.638473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:39.638333    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:40.447477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:40.447024    5104 generic.go:356] "Generic (PLEG): container finished" podID="88be93e3-58b9-4416-87f9-daefc7b22ac0" containerID="1c66316f3b436e4a41d8c6007e2017cb22a20a194817e7800f3c2bd6074952b8" exitCode=0
Oct 30 01:11:40.447477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:40.447065    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x" event={"ID":"88be93e3-58b9-4416-87f9-daefc7b22ac0","Type":"ContainerDied","Data":"1c66316f3b436e4a41d8c6007e2017cb22a20a194817e7800f3c2bd6074952b8"}
Oct 30 01:11:40.638858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:40.638521    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:40.638858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:40.638603    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:41.155480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.147275    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:11:41.638077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.638045    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:11:41.638597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.638099    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 01:11:41.708385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.708315    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:41.834506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.834467    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-xks-ca\") pod \"88be93e3-58b9-4416-87f9-daefc7b22ac0\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") "
Oct 30 01:11:41.834722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.834520    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88be93e3-58b9-4416-87f9-daefc7b22ac0-kube-api-access-w2b6k\" (UniqueName: \"kubernetes.io/projected/88be93e3-58b9-4416-87f9-daefc7b22ac0-kube-api-access-w2b6k\") pod \"88be93e3-58b9-4416-87f9-daefc7b22ac0\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") "
Oct 30 01:11:41.834941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.834917    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-openshift-ca\") pod \"88be93e3-58b9-4416-87f9-daefc7b22ac0\" (UID: \"88be93e3-58b9-4416-87f9-daefc7b22ac0\") "
Oct 30 01:11:41.835303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.835279    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-openshift-ca" pod "88be93e3-58b9-4416-87f9-daefc7b22ac0" (UID: "88be93e3-58b9-4416-87f9-daefc7b22ac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:41.835630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.835611    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:41.836798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.836776    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-xks-ca" pod "88be93e3-58b9-4416-87f9-daefc7b22ac0" (UID: "88be93e3-58b9-4416-87f9-daefc7b22ac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:41.836890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.836849    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88be93e3-58b9-4416-87f9-daefc7b22ac0-kube-api-access-w2b6k" pod "88be93e3-58b9-4416-87f9-daefc7b22ac0" (UID: "88be93e3-58b9-4416-87f9-daefc7b22ac0"). InnerVolumeSpecName "kube-api-access-w2b6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:41.937614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.937580    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88be93e3-58b9-4416-87f9-daefc7b22ac0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:41.937815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:41.937631    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2b6k\" (UniqueName: \"kubernetes.io/projected/88be93e3-58b9-4416-87f9-daefc7b22ac0-kube-api-access-w2b6k\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:11:42.468719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:42.468689    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"
Oct 30 01:11:42.468919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:42.468677    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x" event={"ID":"88be93e3-58b9-4416-87f9-daefc7b22ac0","Type":"ContainerDied","Data":"5ff8729c9c27d3eff2035be4940096480cd62d8acf54ac8a1c1e8f4caee0bc89"}
Oct 30 01:11:42.468919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:42.468860    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff8729c9c27d3eff2035be4940096480cd62d8acf54ac8a1c1e8f4caee0bc89"
Oct 30 01:11:42.644501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:42.644466    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:11:42.644975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:42.644900    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:11:42.645924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:42.645905    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:11:46.470838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:46.470692    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"]
Oct 30 01:11:46.519299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:46.519178    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da60de-qs4wq"]
Oct 30 01:11:46.694825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:46.694792    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d642b1d9-3425-43e0-9e88-70009b08a4f8" path="/var/lib/kubelet/pods/d642b1d9-3425-43e0-9e88-70009b08a4f8/volumes"
Oct 30 01:11:47.545434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:47.545397    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"]
Oct 30 01:11:47.579012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:47.578976    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"]
Oct 30 01:11:52.546953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:52.546866    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" podUID="5775e7cb-9494-47be-b959-4a0cc2a63356" containerName="payload-processing" containerID="cri-o://61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703" gracePeriod=25
Oct 30 01:11:52.579492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:52.579408    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" podUID="f627cdcb-aed4-496b-9137-2a8a60c48827" containerName="payload-pre-processing" containerID="cri-o://219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5" gracePeriod=25
Oct 30 01:11:56.325148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.323970    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"]
Oct 30 01:11:56.325148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.324910    5104 cpu_manager.go:414] "containerMap: removing container" podUID="88be93e3-58b9-4416-87f9-daefc7b22ac0" containerName="revoke-keys"
Oct 30 01:11:56.325148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.324927    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88be93e3-58b9-4416-87f9-daefc7b22ac0" containerName="revoke-keys"
Oct 30 01:11:56.325148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.325138    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="88be93e3-58b9-4416-87f9-daefc7b22ac0" containerName="revoke-keys"
Oct 30 01:11:56.329370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.329347    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.358831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.358791    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-34e896-openshift-default-dockercfg-6zq9h\""
Oct 30 01:11:56.366181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.366144    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"]
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.427993    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4d865c-5038-4871-8e4f-12cac274ca02-istiod-ca-cert\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428098    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-socket\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428123    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-data\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428168    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-podinfo\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428222    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-certs\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428277    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-envoy\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428310    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcsxh\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-kube-api-access-rcsxh\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428388    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-credential-socket\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.434482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.428604    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-token\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.535695    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-token\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.535921    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4d865c-5038-4871-8e4f-12cac274ca02-istiod-ca-cert\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.535988    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-socket\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.536014    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-data\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.536057    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-podinfo\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.536095    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-certs\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.536136    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-envoy\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.536172    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcsxh\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-kube-api-access-rcsxh\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.536232    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-credential-socket\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.536802    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-credential-socket\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.538177    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4d865c-5038-4871-8e4f-12cac274ca02-istiod-ca-cert\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.539559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.538227    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-socket\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.567975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.567940    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-data\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.613623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.538447    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-certs\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.662500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.662344    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-token\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.664206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.664180    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-podinfo\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.690535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.682869    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcsxh\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-kube-api-access-rcsxh\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.743671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.733922    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-envoy\") pod \"e2e-inf-b-34e896-openshift-default-6c7475f655-grr54\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") " pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:56.967480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:56.962996    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:57.855201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:57.855160    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"]
Oct 30 01:11:57.889538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:57.888973    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:57.889538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:57.889087    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:57.889538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:57.889135    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:11:58.312097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:58.312052    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" event={"ID":"2b4d865c-5038-4871-8e4f-12cac274ca02","Type":"ContainerStarted","Data":"fedc832c0aa6907b55353184d1a82bfb260f71c7c32e406aa568c347167d10e7"}
Oct 30 01:11:58.312309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:58.312119    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" event={"ID":"2b4d865c-5038-4871-8e4f-12cac274ca02","Type":"ContainerStarted","Data":"3a522a955fb2ee5a8d99abdc31375a80fa87f33aa4d4cd1db094c9475ff6c2c3"}
Oct 30 01:11:58.377703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:58.377647    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podStartSLOduration=2.377631981 podStartE2EDuration="2.377631981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:58.36592626 +0000 UTC m=+1759.913616755" watchObservedRunningTime="2026-10-30 01:11:58.377631981 +0000 UTC m=+1759.925322477"
Oct 30 01:11:58.970692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:58.970660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:58.976544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:58.976515    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:59.370533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:59.360765    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:59.442477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:59.437020    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:11:59.481479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:59.474594    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" podUID="f627cdcb-aed4-496b-9137-2a8a60c48827" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:11:59.498485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:11:59.493760    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" podUID="5775e7cb-9494-47be-b959-4a0cc2a63356" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:01.223056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.222805    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"]
Oct 30 01:12:01.228143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.228119    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.256716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.256683    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50d078\""
Oct 30 01:12:01.264230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.264196    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"]
Oct 30 01:12:01.265488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.265450    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50d078-dockercfg-bh54d\""
Oct 30 01:12:01.315537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.315508    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"]
Oct 30 01:12:01.323765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.323734    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:01.328982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.328946    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079b3bb9-71bb-4c5a-9bf9-7d509162e665-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.329166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.328987    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xxs\" (UniqueName: \"kubernetes.io/projected/079b3bb9-71bb-4c5a-9bf9-7d509162e665-kube-api-access-78xxs\") pod \"payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.337019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.336077    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"]
Oct 30 01:12:01.498711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.472869    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab79d576-7543-4573-bfca-2cfa6a6000d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:01.498711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.473003    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079b3bb9-71bb-4c5a-9bf9-7d509162e665-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.498711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.473042    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xxs\" (UniqueName: \"kubernetes.io/projected/079b3bb9-71bb-4c5a-9bf9-7d509162e665-kube-api-access-78xxs\") pod \"payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.498711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.473250    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zlch\" (UniqueName: \"kubernetes.io/projected/ab79d576-7543-4573-bfca-2cfa6a6000d6-kube-api-access-9zlch\") pod \"payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:01.498711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.474722    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079b3bb9-71bb-4c5a-9bf9-7d509162e665-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.579927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.579890    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zlch\" (UniqueName: \"kubernetes.io/projected/ab79d576-7543-4573-bfca-2cfa6a6000d6-kube-api-access-9zlch\") pod \"payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:01.580576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.580542    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab79d576-7543-4573-bfca-2cfa6a6000d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:01.632498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.597321    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab79d576-7543-4573-bfca-2cfa6a6000d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:01.699508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.698498    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xxs\" (UniqueName: \"kubernetes.io/projected/079b3bb9-71bb-4c5a-9bf9-7d509162e665-kube-api-access-78xxs\") pod \"payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.706611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.705680    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zlch\" (UniqueName: \"kubernetes.io/projected/ab79d576-7543-4573-bfca-2cfa6a6000d6-kube-api-access-9zlch\") pod \"payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:01.893562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.888679    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:01.957003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:01.956938    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:02.448577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:02.448545    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"]
Oct 30 01:12:02.563606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:02.563574    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"]
Oct 30 01:12:02.563881 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:12:02.563855    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab79d576_7543_4573_bfca_2cfa6a6000d6.slice/crio-daf804abfc88bc2e73e030497110943855d8f11c96a50779dd1850c963f83bb4 WatchSource:0}: Error finding container daf804abfc88bc2e73e030497110943855d8f11c96a50779dd1850c963f83bb4: Status 404 returned error can't find the container with id daf804abfc88bc2e73e030497110943855d8f11c96a50779dd1850c963f83bb4
Oct 30 01:12:02.634163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:02.634119    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" event={"ID":"079b3bb9-71bb-4c5a-9bf9-7d509162e665","Type":"ContainerStarted","Data":"ecad7a2f352dbbdc10be1573403b11791fb719b40b495ca83af4651b8a9fa8b8"}
Oct 30 01:12:02.637978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:02.637948    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz" event={"ID":"ab79d576-7543-4573-bfca-2cfa6a6000d6","Type":"ContainerStarted","Data":"daf804abfc88bc2e73e030497110943855d8f11c96a50779dd1850c963f83bb4"}
Oct 30 01:12:03.414394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.414339    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"]
Oct 30 01:12:03.423869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.423841    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:03.440492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.438749    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"]
Oct 30 01:12:03.440492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.439110    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-34e896\""
Oct 30 01:12:03.465480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.463028    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-34e896-dockercfg-w6swz\""
Oct 30 01:12:03.465480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.465107    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbsb\" (UniqueName: \"kubernetes.io/projected/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-kube-api-access-5zbsb\") pod \"payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:03.465480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.465246    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:03.529479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.524762    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"]
Oct 30 01:12:03.568485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.567066    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbsb\" (UniqueName: \"kubernetes.io/projected/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-kube-api-access-5zbsb\") pod \"payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:03.568485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.567209    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:03.568485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.568332    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:03.874481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.865974    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"]
Oct 30 01:12:03.874481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.868527    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:03.922119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.921622    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zg7q\" (UniqueName: \"kubernetes.io/projected/de1e8764-ed7d-4878-9b31-d2aae18204fd-kube-api-access-6zg7q\") pod \"payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:03.922119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.921887    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1e8764-ed7d-4878-9b31-d2aae18204fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:03.974066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:03.965498    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbsb\" (UniqueName: \"kubernetes.io/projected/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-kube-api-access-5zbsb\") pod \"payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:04.054575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.048692    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zg7q\" (UniqueName: \"kubernetes.io/projected/de1e8764-ed7d-4878-9b31-d2aae18204fd-kube-api-access-6zg7q\") pod \"payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:04.055351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.055323    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1e8764-ed7d-4878-9b31-d2aae18204fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:04.057156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.056384    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1e8764-ed7d-4878-9b31-d2aae18204fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:04.100479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.073430    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" event={"ID":"079b3bb9-71bb-4c5a-9bf9-7d509162e665","Type":"ContainerStarted","Data":"6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335"}
Oct 30 01:12:04.146847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.146696    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:04.166416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.166349    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zg7q\" (UniqueName: \"kubernetes.io/projected/de1e8764-ed7d-4878-9b31-d2aae18204fd-kube-api-access-6zg7q\") pod \"payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:04.167297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.167265    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:04.215820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.215781    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz" event={"ID":"ab79d576-7543-4573-bfca-2cfa6a6000d6","Type":"ContainerStarted","Data":"c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1"}
Oct 30 01:12:04.294902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.285565    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:04.367482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.365856    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:04.438210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.425598    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" podStartSLOduration=3.425576978 podStartE2EDuration="3.425576978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:04.167008847 +0000 UTC m=+1765.714699548" watchObservedRunningTime="2026-10-30 01:12:04.425576978 +0000 UTC m=+1765.973267494"
Oct 30 01:12:04.444172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:04.437385    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz" podStartSLOduration=3.437364877 podStartE2EDuration="3.437364877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:04.362865146 +0000 UTC m=+1765.910555642" watchObservedRunningTime="2026-10-30 01:12:04.437364877 +0000 UTC m=+1765.985055409"
Oct 30 01:12:05.004810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:05.004780    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"]
Oct 30 01:12:05.008816 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:12:05.008787    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1e8764_ed7d_4878_9b31_d2aae18204fd.slice/crio-d2f7ff56a411c087ae3855322b5de59f50447f865910c69a2979f64cd945956e WatchSource:0}: Error finding container d2f7ff56a411c087ae3855322b5de59f50447f865910c69a2979f64cd945956e: Status 404 returned error can't find the container with id d2f7ff56a411c087ae3855322b5de59f50447f865910c69a2979f64cd945956e
Oct 30 01:12:05.015117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:05.015086    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"]
Oct 30 01:12:05.019865 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:12:05.019835    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4de0ed_a83f_4e33_9c10_74ab1d125ac9.slice/crio-0a4632880437be73bfe917b401b6c7833dde27459ca39302cd241c746c502581 WatchSource:0}: Error finding container 0a4632880437be73bfe917b401b6c7833dde27459ca39302cd241c746c502581: Status 404 returned error can't find the container with id 0a4632880437be73bfe917b401b6c7833dde27459ca39302cd241c746c502581
Oct 30 01:12:05.227975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:05.227946    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" event={"ID":"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9","Type":"ContainerStarted","Data":"0a4632880437be73bfe917b401b6c7833dde27459ca39302cd241c746c502581"}
Oct 30 01:12:05.230194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:05.230054    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q" event={"ID":"de1e8764-ed7d-4878-9b31-d2aae18204fd","Type":"ContainerStarted","Data":"0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb"}
Oct 30 01:12:05.230194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:05.230103    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q" event={"ID":"de1e8764-ed7d-4878-9b31-d2aae18204fd","Type":"ContainerStarted","Data":"d2f7ff56a411c087ae3855322b5de59f50447f865910c69a2979f64cd945956e"}
Oct 30 01:12:05.230194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:05.230153    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:05.269896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:05.268828    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q" podStartSLOduration=2.268812405 podStartE2EDuration="2.268812405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:05.266612293 +0000 UTC m=+1766.814302790" watchObservedRunningTime="2026-10-30 01:12:05.268812405 +0000 UTC m=+1766.816502901"
Oct 30 01:12:06.241125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:06.241085    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" event={"ID":"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9","Type":"ContainerStarted","Data":"4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4"}
Oct 30 01:12:06.241634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:06.241158    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:06.292484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:06.283785    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" podStartSLOduration=3.28376251 podStartE2EDuration="3.28376251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:06.28224122 +0000 UTC m=+1767.829931717" watchObservedRunningTime="2026-10-30 01:12:06.28376251 +0000 UTC m=+1767.831453006"
Oct 30 01:12:09.146774 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:09.146735    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:12:09.423217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:09.423178    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" podUID="f627cdcb-aed4-496b-9137-2a8a60c48827" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:09.423513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:09.423174    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" podUID="5775e7cb-9494-47be-b959-4a0cc2a63356" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:11.812546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:11.812503    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"]
Oct 30 01:12:11.871015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:11.870965    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3204e-zrl9x"]
Oct 30 01:12:12.395639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.395592    5104 generic.go:356] "Generic (PLEG): container finished" podID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerID="44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3" exitCode=0
Oct 30 01:12:12.395873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.395650    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" event={"ID":"c8de5c9d-68d2-4468-be09-b9eee683918d","Type":"ContainerDied","Data":"44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3"}
Oct 30 01:12:12.396183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.396160    5104 scope.go:122] "RemoveContainer" containerID="44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3"
Oct 30 01:12:12.398387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.398353    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:12.398525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.398431    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:12.398525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.398481    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:12.603517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.603449    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88be93e3-58b9-4416-87f9-daefc7b22ac0" path="/var/lib/kubelet/pods/88be93e3-58b9-4416-87f9-daefc7b22ac0/volumes"
Oct 30 01:12:12.931756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:12.931685    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:12:13.038140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.038100    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5775e7cb-9494-47be-b959-4a0cc2a63356-kube-api-access-sfc5s\" (UniqueName: \"kubernetes.io/projected/5775e7cb-9494-47be-b959-4a0cc2a63356-kube-api-access-sfc5s\") pod \"5775e7cb-9494-47be-b959-4a0cc2a63356\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") "
Oct 30 01:12:13.038531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.038499    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5775e7cb-9494-47be-b959-4a0cc2a63356-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775e7cb-9494-47be-b959-4a0cc2a63356-plugins-config-volume\") pod \"5775e7cb-9494-47be-b959-4a0cc2a63356\" (UID: \"5775e7cb-9494-47be-b959-4a0cc2a63356\") "
Oct 30 01:12:13.038913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.038885    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5775e7cb-9494-47be-b959-4a0cc2a63356-plugins-config-volume" pod "5775e7cb-9494-47be-b959-4a0cc2a63356" (UID: "5775e7cb-9494-47be-b959-4a0cc2a63356"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:13.039364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.039340    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775e7cb-9494-47be-b959-4a0cc2a63356-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.041432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.041396    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5775e7cb-9494-47be-b959-4a0cc2a63356-kube-api-access-sfc5s" pod "5775e7cb-9494-47be-b959-4a0cc2a63356" (UID: "5775e7cb-9494-47be-b959-4a0cc2a63356"). InnerVolumeSpecName "kube-api-access-sfc5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:13.070632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.070605    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:12:13.146483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.142242    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfc5s\" (UniqueName: \"kubernetes.io/projected/5775e7cb-9494-47be-b959-4a0cc2a63356-kube-api-access-sfc5s\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.243017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.242964    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f627cdcb-aed4-496b-9137-2a8a60c48827-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f627cdcb-aed4-496b-9137-2a8a60c48827-plugins-config-volume\") pod \"f627cdcb-aed4-496b-9137-2a8a60c48827\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") "
Oct 30 01:12:13.243334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.243320    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f627cdcb-aed4-496b-9137-2a8a60c48827-kube-api-access-l4c78\" (UniqueName: \"kubernetes.io/projected/f627cdcb-aed4-496b-9137-2a8a60c48827-kube-api-access-l4c78\") pod \"f627cdcb-aed4-496b-9137-2a8a60c48827\" (UID: \"f627cdcb-aed4-496b-9137-2a8a60c48827\") "
Oct 30 01:12:13.243390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.243361    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f627cdcb-aed4-496b-9137-2a8a60c48827-plugins-config-volume" pod "f627cdcb-aed4-496b-9137-2a8a60c48827" (UID: "f627cdcb-aed4-496b-9137-2a8a60c48827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:13.243865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.243846    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f627cdcb-aed4-496b-9137-2a8a60c48827-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.245650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.245622    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f627cdcb-aed4-496b-9137-2a8a60c48827-kube-api-access-l4c78" pod "f627cdcb-aed4-496b-9137-2a8a60c48827" (UID: "f627cdcb-aed4-496b-9137-2a8a60c48827"). InnerVolumeSpecName "kube-api-access-l4c78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:13.346914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.346797    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4c78\" (UniqueName: \"kubernetes.io/projected/f627cdcb-aed4-496b-9137-2a8a60c48827-kube-api-access-l4c78\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.423507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.423441    5104 generic.go:356] "Generic (PLEG): container finished" podID="5775e7cb-9494-47be-b959-4a0cc2a63356" containerID="61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703" exitCode=0
Oct 30 01:12:13.423730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.423692    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" event={"ID":"5775e7cb-9494-47be-b959-4a0cc2a63356","Type":"ContainerDied","Data":"61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703"}
Oct 30 01:12:13.423813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.423773    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z" event={"ID":"5775e7cb-9494-47be-b959-4a0cc2a63356","Type":"ContainerDied","Data":"b47fb7b4a46e9fbc2bae715f87da9ec6f82d1e32eb630c323838be1acc6abffc"}
Oct 30 01:12:13.423813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.423796    5104 scope.go:122] "RemoveContainer" containerID="61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703"
Oct 30 01:12:13.423954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.423928    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"
Oct 30 01:12:13.510095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.510063    5104 scope.go:122] "RemoveContainer" containerID="61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703"
Oct 30 01:12:13.511718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.511689    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" event={"ID":"c8de5c9d-68d2-4468-be09-b9eee683918d","Type":"ContainerStarted","Data":"d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6"}
Oct 30 01:12:13.513372 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:13.513341    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703\": container with ID starting with 61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703 not found: ID does not exist" containerID="61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703"
Oct 30 01:12:13.524596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.524535    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703"} err="failed to get container status \"61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703\": rpc error: code = NotFound desc = could not find container \"61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703\": container with ID starting with 61c70be96695719ca34caa124242c551a9bc4e478c4a0ab243b90ba5610b8703 not found: ID does not exist"
Oct 30 01:12:13.524924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.521080    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" event={"ID":"f627cdcb-aed4-496b-9137-2a8a60c48827","Type":"ContainerDied","Data":"219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5"}
Oct 30 01:12:13.525100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.525084    5104 scope.go:122] "RemoveContainer" containerID="219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5"
Oct 30 01:12:13.525313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.521047    5104 generic.go:356] "Generic (PLEG): container finished" podID="f627cdcb-aed4-496b-9137-2a8a60c48827" containerID="219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5" exitCode=0
Oct 30 01:12:13.525527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.525506    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8" event={"ID":"f627cdcb-aed4-496b-9137-2a8a60c48827","Type":"ContainerDied","Data":"35a530545c4affc829161d31a7aecfafc5f37b71f745fafbeb9ec4a52fbecc95"}
Oct 30 01:12:13.525653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.521175    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"
Oct 30 01:12:13.762108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.740618    5104 scope.go:122] "RemoveContainer" containerID="219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5"
Oct 30 01:12:13.763288 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:13.763248    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5\": container with ID starting with 219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5 not found: ID does not exist" containerID="219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5"
Oct 30 01:12:13.763616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.763567    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5"} err="failed to get container status \"219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5\": rpc error: code = NotFound desc = could not find container \"219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5\": container with ID starting with 219df8f036b2d65591f9a8984f1c3eb79a54d91818e5f33c8148964ba707e8c5 not found: ID does not exist"
Oct 30 01:12:13.768094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.768068    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"]
Oct 30 01:12:13.823896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.773341    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3204e-756d475586-r7c4z"]
Oct 30 01:12:13.826686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.826578    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:12:13.827984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.827955    5104 patch_prober.go:29] interesting pod/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:12:13.828202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.828177    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 01:12:13.839548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.839517    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"]
Oct 30 01:12:13.847572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:13.847540    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3204e-58597f8cf5-tczb8"]
Oct 30 01:12:14.611545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:14.611507    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5775e7cb-9494-47be-b959-4a0cc2a63356" path="/var/lib/kubelet/pods/5775e7cb-9494-47be-b959-4a0cc2a63356/volumes"
Oct 30 01:12:14.612845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:14.612821    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f627cdcb-aed4-496b-9137-2a8a60c48827" path="/var/lib/kubelet/pods/f627cdcb-aed4-496b-9137-2a8a60c48827/volumes"
Oct 30 01:12:14.827389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:14.827356    5104 patch_prober.go:29] interesting pod/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:12:14.827608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:14.827411    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 01:12:15.233242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:15.233211    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:12:15.233502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:15.233483    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:12:15.826990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:15.826954    5104 patch_prober.go:29] interesting pod/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:12:15.827404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:15.827022    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 01:12:16.244907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:16.244875    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:12:16.827096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:16.827056    5104 patch_prober.go:29] interesting pod/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:12:16.827543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:16.827108    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 01:12:17.281479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:17.280730    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:12:17.493482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:17.489567    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:12:17.827601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:17.827555    5104 patch_prober.go:29] interesting pod/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:12:17.828038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:17.827651    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 01:12:18.399480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:18.397149    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:12:18.821506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:18.821404    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"]
Oct 30 01:12:18.823069 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:18.822981    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" containerID="cri-o://0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86" gracePeriod=30
Oct 30 01:12:18.853484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:18.847406    5104 patch_prober.go:29] interesting pod/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:12:18.853484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:18.847494    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 30 01:12:19.303237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.302565    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"]
Oct 30 01:12:19.315725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.315675    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5775e7cb-9494-47be-b959-4a0cc2a63356" containerName="payload-processing"
Oct 30 01:12:19.315933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.315916    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5775e7cb-9494-47be-b959-4a0cc2a63356" containerName="payload-processing"
Oct 30 01:12:19.316109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.316097    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f627cdcb-aed4-496b-9137-2a8a60c48827" containerName="payload-pre-processing"
Oct 30 01:12:19.316337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.316321    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f627cdcb-aed4-496b-9137-2a8a60c48827" containerName="payload-pre-processing"
Oct 30 01:12:19.316843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.316820    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5775e7cb-9494-47be-b959-4a0cc2a63356" containerName="payload-processing"
Oct 30 01:12:19.316992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.316979    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f627cdcb-aed4-496b-9137-2a8a60c48827" containerName="payload-pre-processing"
Oct 30 01:12:19.368479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.356793    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.378825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.378791    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:19.380070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.380047    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:19.400808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.388085    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.400808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.388376    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56rl\" (UniqueName: \"kubernetes.io/projected/b4f07832-2dfc-4a31-9677-564beb843a44-kube-api-access-z56rl\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.400808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.388424    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.425193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.416617    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:12:19.425193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.416687    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:19.491487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.490235    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.491487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.490500    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.491487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.490802    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56rl\" (UniqueName: \"kubernetes.io/projected/b4f07832-2dfc-4a31-9677-564beb843a44-kube-api-access-z56rl\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.503491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.501776    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.528477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.522953    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.543617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.540528    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"]
Oct 30 01:12:19.559330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.559257    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56rl\" (UniqueName: \"kubernetes.io/projected/b4f07832-2dfc-4a31-9677-564beb843a44-kube-api-access-z56rl\") pod \"maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.765665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.765627    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:19.968256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.967959    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:12:19.970839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:19.970776    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:12:20.112863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:20.112828    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"]
Oct 30 01:12:20.797491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:20.794913    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9" event={"ID":"b4f07832-2dfc-4a31-9677-564beb843a44","Type":"ContainerStarted","Data":"418f7e4e604f03118951bb44f4ce2e7539480fa5631af711ebcae1caa7b4f314"}
Oct 30 01:12:21.569293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:21.569255    5104 patch_prober.go:29] interesting pod/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:12:21.569709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:21.569320    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:12:21.808091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:21.807995    5104 generic.go:356] "Generic (PLEG): container finished" podID="b4f07832-2dfc-4a31-9677-564beb843a44" containerID="5edc3708cb59cc62fe7766eafc1754ccc7eed7be37fe3034d0b5c541d6236801" exitCode=0
Oct 30 01:12:21.808091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:21.808073    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9" event={"ID":"b4f07832-2dfc-4a31-9677-564beb843a44","Type":"ContainerDied","Data":"5edc3708cb59cc62fe7766eafc1754ccc7eed7be37fe3034d0b5c541d6236801"}
Oct 30 01:12:23.065657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.065627    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:23.268748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.268698    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f07832-2dfc-4a31-9677-564beb843a44-kube-api-access-z56rl\" (UniqueName: \"kubernetes.io/projected/b4f07832-2dfc-4a31-9677-564beb843a44-kube-api-access-z56rl\") pod \"b4f07832-2dfc-4a31-9677-564beb843a44\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") "
Oct 30 01:12:23.269768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.269742    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-xks-ca\") pod \"b4f07832-2dfc-4a31-9677-564beb843a44\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") "
Oct 30 01:12:23.275695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.275649    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-openshift-ca\") pod \"b4f07832-2dfc-4a31-9677-564beb843a44\" (UID: \"b4f07832-2dfc-4a31-9677-564beb843a44\") "
Oct 30 01:12:23.276369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.276252    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-openshift-ca" pod "b4f07832-2dfc-4a31-9677-564beb843a44" (UID: "b4f07832-2dfc-4a31-9677-564beb843a44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:23.302740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.302700    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:23.329975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.329936    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f07832-2dfc-4a31-9677-564beb843a44-kube-api-access-z56rl" pod "b4f07832-2dfc-4a31-9677-564beb843a44" (UID: "b4f07832-2dfc-4a31-9677-564beb843a44"). InnerVolumeSpecName "kube-api-access-z56rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:23.331010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.330968    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-xks-ca" pod "b4f07832-2dfc-4a31-9677-564beb843a44" (UID: "b4f07832-2dfc-4a31-9677-564beb843a44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:23.405237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.405191    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f07832-2dfc-4a31-9677-564beb843a44-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:23.405564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.405532    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z56rl\" (UniqueName: \"kubernetes.io/projected/b4f07832-2dfc-4a31-9677-564beb843a44-kube-api-access-z56rl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:23.831171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.831141    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"
Oct 30 01:12:23.831365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.831139    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9" event={"ID":"b4f07832-2dfc-4a31-9677-564beb843a44","Type":"ContainerDied","Data":"418f7e4e604f03118951bb44f4ce2e7539480fa5631af711ebcae1caa7b4f314"}
Oct 30 01:12:23.831365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:23.831273    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="418f7e4e604f03118951bb44f4ce2e7539480fa5631af711ebcae1caa7b4f314"
Oct 30 01:12:24.442371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.442340    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:12:24.528570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.528516    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-podinfo\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.528782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.528674    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-envoy\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.528782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.528750    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-kube-api-access-psk4k\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-kube-api-access-psk4k\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.528901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.528885    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-certs\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.529093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.529072    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-socket\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.529161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.529143    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-data\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.529221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.529197    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42f8ae0-21a7-446c-95f0-06b568a26d87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42f8ae0-21a7-446c-95f0-06b568a26d87-istiod-ca-cert\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.529277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.529228    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-token\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-token\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.529358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.529341    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-credential-socket\") pod \"d42f8ae0-21a7-446c-95f0-06b568a26d87\" (UID: \"d42f8ae0-21a7-446c-95f0-06b568a26d87\") "
Oct 30 01:12:24.529768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.529741    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-certs" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:24.529987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.529959    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-socket" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:24.530327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.530302    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42f8ae0-21a7-446c-95f0-06b568a26d87-istiod-ca-cert" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:24.531641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.531594    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-credential-socket" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:24.531809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.531787    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-data" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:24.533164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.533136    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-podinfo" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:24.534578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.534542    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-envoy" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:24.535419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.535390    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-kube-api-access-psk4k" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "kube-api-access-psk4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:24.536435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536408    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.536656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536440    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.536656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536515    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psk4k\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-kube-api-access-psk4k\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.536656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536532    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.536656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536546    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.536656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536559    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.536656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536572    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42f8ae0-21a7-446c-95f0-06b568a26d87-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.536656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.536585    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42f8ae0-21a7-446c-95f0-06b568a26d87-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.538328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.538299    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-token" pod "d42f8ae0-21a7-446c-95f0-06b568a26d87" (UID: "d42f8ae0-21a7-446c-95f0-06b568a26d87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:24.637992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.637955    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42f8ae0-21a7-446c-95f0-06b568a26d87-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:24.843917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.843880    5104 generic.go:356] "Generic (PLEG): container finished" podID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerID="0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86" exitCode=0
Oct 30 01:12:24.844128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.843922    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" event={"ID":"d42f8ae0-21a7-446c-95f0-06b568a26d87","Type":"ContainerDied","Data":"0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86"}
Oct 30 01:12:24.844128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.843946    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"
Oct 30 01:12:24.844128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.843983    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5" event={"ID":"d42f8ae0-21a7-446c-95f0-06b568a26d87","Type":"ContainerDied","Data":"465bd0daa5f4cc66082a9dfa83da754654b5602bac77271512b8cc7fd9e2ab7c"}
Oct 30 01:12:24.844128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.844023    5104 scope.go:122] "RemoveContainer" containerID="0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86"
Oct 30 01:12:24.856275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.856253    5104 scope.go:122] "RemoveContainer" containerID="ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39"
Oct 30 01:12:24.867743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.867717    5104 scope.go:122] "RemoveContainer" containerID="0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86"
Oct 30 01:12:24.868066 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:24.868041    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86\": container with ID starting with 0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86 not found: ID does not exist" containerID="0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86"
Oct 30 01:12:24.868122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.868079    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86"} err="failed to get container status \"0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86\": rpc error: code = NotFound desc = could not find container \"0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86\": container with ID starting with 0ea39de60e6709f7f7aa5b79f6e414f7ce8ce8263467a1f2b3ad941d2dc74c86 not found: ID does not exist"
Oct 30 01:12:24.868122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.868106    5104 scope.go:122] "RemoveContainer" containerID="ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39"
Oct 30 01:12:24.868379 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:24.868358    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39\": container with ID starting with ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39 not found: ID does not exist" containerID="ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39"
Oct 30 01:12:24.868510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.868386    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39"} err="failed to get container status \"ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39\": rpc error: code = NotFound desc = could not find container \"ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39\": container with ID starting with ba8349d360ae6e48b15f24d1aa7db383232b9b204fc1a488d24b5140aec2af39 not found: ID does not exist"
Oct 30 01:12:24.907243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.905136    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"]
Oct 30 01:12:24.910886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:24.910849    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a3204e-openshift-default-57fcb546f7-pfmc5"]
Oct 30 01:12:26.582659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:26.582619    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" path="/var/lib/kubelet/pods/d42f8ae0-21a7-446c-95f0-06b568a26d87/volumes"
Oct 30 01:12:29.695474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:29.695422    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"]
Oct 30 01:12:29.727679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:29.727639    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"]
Oct 30 01:12:31.980961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:31.980930    5104 generic.go:356] "Generic (PLEG): container finished" podID="a69c9213-1688-410c-8e09-06ac2dc53702" containerID="0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2" exitCode=0
Oct 30 01:12:31.981437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:31.981000    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" event={"ID":"a69c9213-1688-410c-8e09-06ac2dc53702","Type":"ContainerDied","Data":"0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2"}
Oct 30 01:12:31.981559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:31.981500    5104 scope.go:122] "RemoveContainer" containerID="0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2"
Oct 30 01:12:31.983384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:31.983357    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:31.983519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:31.983423    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:31.983519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:31.983493    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:32.992630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:32.992590    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" event={"ID":"a69c9213-1688-410c-8e09-06ac2dc53702","Type":"ContainerStarted","Data":"0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f"}
Oct 30 01:12:33.756479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:33.756126    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:12:33.780485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:33.764585    5104 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:12:33.780485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:33.764649    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:12:34.696610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:34.696556    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls" podUID="4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" containerName="payload-processing" containerID="cri-o://5a366c179f72578357d2ba7d83d9ea8ce9213c736b8dbe9ff41861e84657ece0" gracePeriod=25
Oct 30 01:12:34.743676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:34.743628    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" podUID="a8f0a3cf-9833-409e-a859-de471cc262fb" containerName="payload-pre-processing" containerID="cri-o://44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674" gracePeriod=25
Oct 30 01:12:34.749504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:34.749477    5104 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:12:34.749631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:34.749531    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:12:35.108483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.102591    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:12:35.346511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.327199    5104 generic.go:356] "Generic (PLEG): container finished" podID="4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" containerID="5a366c179f72578357d2ba7d83d9ea8ce9213c736b8dbe9ff41861e84657ece0" exitCode=0
Oct 30 01:12:35.346511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.327539    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls" event={"ID":"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7","Type":"ContainerDied","Data":"5a366c179f72578357d2ba7d83d9ea8ce9213c736b8dbe9ff41861e84657ece0"}
Oct 30 01:12:35.567258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.566764    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:12:35.715489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.713243    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-plugins-config-volume\") pod \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") "
Oct 30 01:12:35.715489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.713428    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-kube-api-access-nvv4v\" (UniqueName: \"kubernetes.io/projected/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-kube-api-access-nvv4v\") pod \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\" (UID: \"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7\") "
Oct 30 01:12:35.787777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.736750    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-plugins-config-volume" pod "4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" (UID: "4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:35.787777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.742087    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-kube-api-access-nvv4v" pod "4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" (UID: "4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7"). InnerVolumeSpecName "kube-api-access-nvv4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:35.787777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.750004    5104 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:12:35.787777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.750058    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:12:35.887684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.887423    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:35.888210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:35.888182    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvv4v\" (UniqueName: \"kubernetes.io/projected/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7-kube-api-access-nvv4v\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:36.671618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:36.671579    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"
Oct 30 01:12:36.744477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:36.741728    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls" event={"ID":"4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7","Type":"ContainerDied","Data":"ef7155ece4596a78ca487edafdd3506d2a01db43dac0728a35c4c9650ef82d5f"}
Oct 30 01:12:36.744477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:36.743064    5104 scope.go:122] "RemoveContainer" containerID="5a366c179f72578357d2ba7d83d9ea8ce9213c736b8dbe9ff41861e84657ece0"
Oct 30 01:12:36.751554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:36.751498    5104 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:12:36.875499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:36.872015    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:12:36.892235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:36.887569    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"]
Oct 30 01:12:36.903504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:36.896916    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e233f-b5cfd5c8-8nwls"]
Oct 30 01:12:37.749781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:37.749746    5104 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:12:37.750236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:37.749809    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:12:38.482243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.475910    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"]
Oct 30 01:12:38.482243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.476164    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" containerID="cri-o://f0ee5e068d57df3ddc8a7ca3b450467d9c14babfa329df065c99e1b2584a40d5" gracePeriod=30
Oct 30 01:12:38.754577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.754309    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" path="/var/lib/kubelet/pods/4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7/volumes"
Oct 30 01:12:38.754577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.754326    5104 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:12:38.754577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.754373    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:12:38.756343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.754807    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"]
Oct 30 01:12:38.756343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.755014    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy" containerID="cri-o://0da5e3cb627ea30a958395f457ce1dbddb5266aeacfc0a7fadd0b6cb68a807bf" gracePeriod=30
Oct 30 01:12:38.838755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.820365    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"]
Oct 30 01:12:38.838755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.820618    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy" containerID="cri-o://5e5f0a671a4f67b6aee40117e7055b4dc70d95bcf749e4943345115897ef36e7" gracePeriod=30
Oct 30 01:12:38.849378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.849215    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"]
Oct 30 01:12:38.854190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.854158    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy" containerID="cri-o://ef71142048cbf7d798d5500d173d1f669985f8f84ba59d33f9a181ea6e674cc4" gracePeriod=30
Oct 30 01:12:38.986480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.985271    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:12:38.992496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:38.991156    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:12:39.016528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:39.007741    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:12:39.016528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:39.014782    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:12:39.142894 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:39.142856    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:12:39.749143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:39.749110    5104 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:12:39.749324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:39.749166    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 01:12:40.756979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:40.756752    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:12:40.766066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:40.765017    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:12:41.256409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:41.256372    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:12:41.256635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:41.256430    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:12:41.602536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:41.601572    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" podUID="a8f0a3cf-9833-409e-a859-de471cc262fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:42.554220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.554187    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"]
Oct 30 01:12:42.555069 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555051    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy"
Oct 30 01:12:42.555069 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555069    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy"
Oct 30 01:12:42.555201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555105    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" containerName="payload-processing"
Oct 30 01:12:42.555201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555114    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" containerName="payload-processing"
Oct 30 01:12:42.555201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555132    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b4f07832-2dfc-4a31-9677-564beb843a44" containerName="revoke-keys"
Oct 30 01:12:42.555201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555139    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f07832-2dfc-4a31-9677-564beb843a44" containerName="revoke-keys"
Oct 30 01:12:42.555201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555153    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy"
Oct 30 01:12:42.555201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555161    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy"
Oct 30 01:12:42.555477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555448    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy"
Oct 30 01:12:42.555528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555488    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2fe939-5dcb-49ca-b8d3-4ef3834e83c7" containerName="payload-processing"
Oct 30 01:12:42.555528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555500    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f07832-2dfc-4a31-9677-564beb843a44" containerName="revoke-keys"
Oct 30 01:12:42.555528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.555509    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42f8ae0-21a7-446c-95f0-06b568a26d87" containerName="istio-proxy"
Oct 30 01:12:42.561275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.561249    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:42.599491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.596942    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e69dee\""
Oct 30 01:12:42.710495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.707636    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"]
Oct 30 01:12:42.784484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.783908    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"]
Oct 30 01:12:42.972510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.970093    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe71f7b-8700-4284-b72d-1af3a263cd4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:42.972510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:42.970271    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krhg6\" (UniqueName: \"kubernetes.io/projected/afe71f7b-8700-4284-b72d-1af3a263cd4c-kube-api-access-krhg6\") pod \"payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:43.135104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.135066    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:43.238545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.228614    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e69dee-dockercfg-vklpx\""
Oct 30 01:12:43.317480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.290007    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe71f7b-8700-4284-b72d-1af3a263cd4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:43.317480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.290337    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krhg6\" (UniqueName: \"kubernetes.io/projected/afe71f7b-8700-4284-b72d-1af3a263cd4c-kube-api-access-krhg6\") pod \"payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:43.331716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.319910    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe71f7b-8700-4284-b72d-1af3a263cd4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:43.422262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.392854    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfspj\" (UniqueName: \"kubernetes.io/projected/7036635b-a1e2-4679-99dc-0e054dbbc2c5-kube-api-access-wfspj\") pod \"payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:43.422262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.392973    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7036635b-a1e2-4679-99dc-0e054dbbc2c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:43.424557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.424527    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"]
Oct 30 01:12:43.506597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.506500    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfspj\" (UniqueName: \"kubernetes.io/projected/7036635b-a1e2-4679-99dc-0e054dbbc2c5-kube-api-access-wfspj\") pod \"payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:43.507383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.507353    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7036635b-a1e2-4679-99dc-0e054dbbc2c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:43.549933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.538915    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krhg6\" (UniqueName: \"kubernetes.io/projected/afe71f7b-8700-4284-b72d-1af3a263cd4c-kube-api-access-krhg6\") pod \"payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:43.550216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.549577    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7036635b-a1e2-4679-99dc-0e054dbbc2c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:43.578481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.560443    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfspj\" (UniqueName: \"kubernetes.io/projected/7036635b-a1e2-4679-99dc-0e054dbbc2c5-kube-api-access-wfspj\") pod \"payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:43.847056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.842833    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:43.913498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:43.858386    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:44.321500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.317740    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:44.321500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.318737    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:44.486482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.483978    5104 generic.go:356] "Generic (PLEG): container finished" podID="bd9114fe-c972-4605-afa7-3975a55d3712" containerID="0da5e3cb627ea30a958395f457ce1dbddb5266aeacfc0a7fadd0b6cb68a807bf" exitCode=0
Oct 30 01:12:44.486482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.484131    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" event={"ID":"bd9114fe-c972-4605-afa7-3975a55d3712","Type":"ContainerDied","Data":"0da5e3cb627ea30a958395f457ce1dbddb5266aeacfc0a7fadd0b6cb68a807bf"}
Oct 30 01:12:44.487522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.486989    5104 generic.go:356] "Generic (PLEG): container finished" podID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerID="5e5f0a671a4f67b6aee40117e7055b4dc70d95bcf749e4943345115897ef36e7" exitCode=0
Oct 30 01:12:44.487522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.487217    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" event={"ID":"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a","Type":"ContainerDied","Data":"5e5f0a671a4f67b6aee40117e7055b4dc70d95bcf749e4943345115897ef36e7"}
Oct 30 01:12:44.487522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.487270    5104 scope.go:122] "RemoveContainer" containerID="5e1d3247a02de3ab22ef77909c2b6b5258322e042ad4492576e5842b4f139477"
Oct 30 01:12:44.493325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.492398    5104 generic.go:356] "Generic (PLEG): container finished" podID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerID="ef71142048cbf7d798d5500d173d1f669985f8f84ba59d33f9a181ea6e674cc4" exitCode=0
Oct 30 01:12:44.493325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.492654    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" event={"ID":"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a","Type":"ContainerDied","Data":"ef71142048cbf7d798d5500d173d1f669985f8f84ba59d33f9a181ea6e674cc4"}
Oct 30 01:12:44.530152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.529969    5104 generic.go:356] "Generic (PLEG): container finished" podID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerID="f0ee5e068d57df3ddc8a7ca3b450467d9c14babfa329df065c99e1b2584a40d5" exitCode=0
Oct 30 01:12:44.530152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.530006    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" event={"ID":"17386f66-3f21-4f9e-8a12-9700848daaf9","Type":"ContainerDied","Data":"f0ee5e068d57df3ddc8a7ca3b450467d9c14babfa329df065c99e1b2584a40d5"}
Oct 30 01:12:44.557721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.557326    5104 scope.go:122] "RemoveContainer" containerID="264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a"
Oct 30 01:12:44.640783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.640544    5104 scope.go:122] "RemoveContainer" containerID="5da5806efe89fd1f41748c986e754e8727ec6d203f064c93d4ca038a3b6da9f4"
Oct 30 01:12:44.683466 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.683338    5104 scope.go:122] "RemoveContainer" containerID="942f44cb56d8954e1dbdac8d57c3bdf69e2068786d230b6d9aaefaaf671e32f4"
Oct 30 01:12:44.961475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.957654    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"]
Oct 30 01:12:44.969482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:44.962681    5104 scope.go:122] "RemoveContainer" containerID="423a6475ab8052271a5c3bda0d19a1b7577567fd6cba8faf9ffae55ef234fad3"
Oct 30 01:12:45.084510 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:12:45.073588    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe71f7b_8700_4284_b72d_1af3a263cd4c.slice/crio-2e53b969ec4b334b6ab754bc50ca6f71be1ae04447bca419976686ce859fa465 WatchSource:0}: Error finding container 2e53b969ec4b334b6ab754bc50ca6f71be1ae04447bca419976686ce859fa465: Status 404 returned error can't find the container with id 2e53b969ec4b334b6ab754bc50ca6f71be1ae04447bca419976686ce859fa465
Oct 30 01:12:45.262071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.261952    5104 scope.go:122] "RemoveContainer" containerID="7b79417619cc5a8c462772fe7192825d359e8703053507cf52bdbcae3386742f"
Oct 30 01:12:45.304380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.304000    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 01:12:45.325664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.325635    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:12:45.328525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.328445    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:12:45.334739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.334641    5104 scope.go:122] "RemoveContainer" containerID="264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a"
Oct 30 01:12:45.339181 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:45.335470    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a\": container with ID starting with 264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a not found: ID does not exist" containerID="264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a"
Oct 30 01:12:45.339181 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:45.335507    5104 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a\": rpc error: code = NotFound desc = could not find container \"264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a\": container with ID starting with 264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a not found: ID does not exist" containerID="264355fa46334de188fe34090be1516d83a18d36a535736ff87c8c136421a15a"
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.423761    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-podinfo\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.423867    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-kube-api-access-qpmkb\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-kube-api-access-qpmkb\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.423906    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istiod-ca-cert\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.423955    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9114fe-c972-4605-afa7-3975a55d3712-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9114fe-c972-4605-afa7-3975a55d3712-istiod-ca-cert\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.423990    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-certs\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.424031    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istiod-ca-cert\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.424059    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-istio-token\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-istio-token\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.424105    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd9114fe-c972-4605-afa7-3975a55d3712-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9114fe-c972-4605-afa7-3975a55d3712-istio-podinfo\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.424219    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-data\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.424247    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-token\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-token\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.424371    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-kube-api-access-5lshq\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-kube-api-access-5lshq\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.424443    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-envoy\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.425831    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istiod-ca-cert" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:45.450564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.432931    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-data" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.515594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.514209    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istiod-ca-cert" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:45.522738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.522697    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-podinfo\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.523047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.523026    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-socket\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.523197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.523182    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-certs\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.523344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.523328    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-credential-socket\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.523558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.523537    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-credential-socket\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.523742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.523726    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-socket\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.523862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.523846    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-envoy\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.523977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.523963    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-socket\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.524087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.524073    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-credential-socket\") pod \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\" (UID: \"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a\") "
Oct 30 01:12:45.524198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.524186    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-certs\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.524313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.524301    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-token\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-token\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.524443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.524430    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-envoy\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.524575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.524561    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-data\") pod \"bd9114fe-c972-4605-afa7-3975a55d3712\" (UID: \"bd9114fe-c972-4605-afa7-3975a55d3712\") "
Oct 30 01:12:45.524700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.524686    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-kube-api-access-zlq7d\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-kube-api-access-zlq7d\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.524836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.524821    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-data\") pod \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\" (UID: \"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a\") "
Oct 30 01:12:45.525977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.525937    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.526125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.526111    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.526220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.526210    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.526563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.526544    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-data" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.552520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.546429    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-socket" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.552520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.546749    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-certs" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.552520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.546934    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-credential-socket" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.552520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.551679    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 01:12:45.569550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.558054    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-credential-socket" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.569550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.561596    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-socket" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.586616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.570719    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9114fe-c972-4605-afa7-3975a55d3712-istiod-ca-cert" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:45.586616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.571044    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-certs" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.586616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.571772    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-envoy" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.639128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.638424    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-token" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.639128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.638691    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-socket" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.639128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.638867    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-credential-socket" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.639128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.639039    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-certs" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.639128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.639094    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-podinfo" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:45.644603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.643764    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd9114fe-c972-4605-afa7-3975a55d3712-istio-podinfo" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:45.644603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.644227    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-kube-api-access-qpmkb" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "kube-api-access-qpmkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.645563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.644719    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-envoy" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.645563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.644974    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-data" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.648036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.647056    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-token" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648799    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648845    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648863    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648881    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648898    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648912    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648925    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648939    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648953    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpmkb\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-kube-api-access-qpmkb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648968    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9114fe-c972-4605-afa7-3975a55d3712-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648982    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.648997    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9114fe-c972-4605-afa7-3975a55d3712-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.649010    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.649026    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.649041    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.649058    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.649073    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.650728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.649087    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.749200    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-podinfo" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.749770    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-podinfo\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.749841    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-socket\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.749879    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-credential-socket\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.749943    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-envoy\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.749978    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-certs\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.750065    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-data\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.750323    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17386f66-3f21-4f9e-8a12-9700848daaf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17386f66-3f21-4f9e-8a12-9700848daaf9-istiod-ca-cert\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.750582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.750408    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-kube-api-access-9tcwz\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-kube-api-access-9tcwz\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.751283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.750451    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-token\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-token\") pod \"17386f66-3f21-4f9e-8a12-9700848daaf9\" (UID: \"17386f66-3f21-4f9e-8a12-9700848daaf9\") "
Oct 30 01:12:45.752350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.752325    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9114fe-c972-4605-afa7-3975a55d3712-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.752604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.752590    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.753640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.753604    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-data" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.753835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.753814    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-certs" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.754317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.754293    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17386f66-3f21-4f9e-8a12-9700848daaf9-istiod-ca-cert" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:45.764704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.763476    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-kube-api-access-zlq7d" pod "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" (UID: "6aa02a4f-e4f4-4b34-b165-e306f42c7c2a"). InnerVolumeSpecName "kube-api-access-zlq7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.764704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.763538    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-kube-api-access-9tcwz" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "kube-api-access-9tcwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.764704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.763630    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-istio-token" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.764704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.763911    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-kube-api-access-5lshq" pod "bd9114fe-c972-4605-afa7-3975a55d3712" (UID: "bd9114fe-c972-4605-afa7-3975a55d3712"). InnerVolumeSpecName "kube-api-access-5lshq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.764704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.764116    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-envoy" pod "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" (UID: "1dfedb57-fd1c-4d6f-9d27-249f3a7f244a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.764704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.764582    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-envoy" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.764704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.764650    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-podinfo" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:45.764995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.764858    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-credential-socket" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.768215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.768189    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-token" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:45.769578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.762859    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-socket" pod "17386f66-3f21-4f9e-8a12-9700848daaf9" (UID: "17386f66-3f21-4f9e-8a12-9700848daaf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:45.841231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.833652    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"]
Oct 30 01:12:45.856372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856339    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17386f66-3f21-4f9e-8a12-9700848daaf9-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.856372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856372    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tcwz\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-kube-api-access-9tcwz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856382    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856396    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lshq\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-kube-api-access-5lshq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856406    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856415    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856423    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856434    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856443    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856469    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856479    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlq7d\" (UniqueName: \"kubernetes.io/projected/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a-kube-api-access-zlq7d\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856490    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17386f66-3f21-4f9e-8a12-9700848daaf9-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.861010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.856499    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9114fe-c972-4605-afa7-3975a55d3712-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:45.958539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.958451    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww" event={"ID":"bd9114fe-c972-4605-afa7-3975a55d3712","Type":"ContainerDied","Data":"df7d6f4590132d934fbfaf4f91a580b1147638b03b9c4fab4e000a9d11196149"}
Oct 30 01:12:45.962521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.958660    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"
Oct 30 01:12:45.962521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.961572    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z" event={"ID":"1dfedb57-fd1c-4d6f-9d27-249f3a7f244a","Type":"ContainerDied","Data":"e0d2c9db3d66c8640753691dc61dea3f454b599db91f3e10e7b83dbce7943b41"}
Oct 30 01:12:45.962521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.961636    5104 scope.go:122] "RemoveContainer" containerID="0da5e3cb627ea30a958395f457ce1dbddb5266aeacfc0a7fadd0b6cb68a807bf"
Oct 30 01:12:45.962521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.961165    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"
Oct 30 01:12:45.970962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.970762    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n" event={"ID":"afe71f7b-8700-4284-b72d-1af3a263cd4c","Type":"ContainerStarted","Data":"2e53b969ec4b334b6ab754bc50ca6f71be1ae04447bca419976686ce859fa465"}
Oct 30 01:12:45.970962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.970809    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:45.977844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.973308    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg" event={"ID":"6aa02a4f-e4f4-4b34-b165-e306f42c7c2a","Type":"ContainerDied","Data":"c4684e29a0b46f28da7f828bea7d817257623c46e009f178486f6b0342f95d40"}
Oct 30 01:12:45.977844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.973437    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"
Oct 30 01:12:45.994280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.993723    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492" event={"ID":"17386f66-3f21-4f9e-8a12-9700848daaf9","Type":"ContainerDied","Data":"afc2d96faf7794072ac3706eb0181045e8b7dc2a6aa1c75c20615b9d7649ab05"}
Oct 30 01:12:45.994280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:45.993867    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"
Oct 30 01:12:46.022797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.022771    5104 scope.go:122] "RemoveContainer" containerID="5e5f0a671a4f67b6aee40117e7055b4dc70d95bcf749e4943345115897ef36e7"
Oct 30 01:12:46.065060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.065032    5104 scope.go:122] "RemoveContainer" containerID="ef71142048cbf7d798d5500d173d1f669985f8f84ba59d33f9a181ea6e674cc4"
Oct 30 01:12:46.082980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.082885    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n" podStartSLOduration=4.0828628160000004 podStartE2EDuration="4.082862816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:46.069891156 +0000 UTC m=+1807.617581654" watchObservedRunningTime="2026-10-30 01:12:46.082862816 +0000 UTC m=+1807.630553313"
Oct 30 01:12:46.115033 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.114957    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"]
Oct 30 01:12:46.123092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.123070    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-2gm4z"]
Oct 30 01:12:46.123942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.123917    5104 scope.go:122] "RemoveContainer" containerID="f0ee5e068d57df3ddc8a7ca3b450467d9c14babfa329df065c99e1b2584a40d5"
Oct 30 01:12:46.156876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.156848    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"]
Oct 30 01:12:46.163967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.163931    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-5f492"]
Oct 30 01:12:46.346278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.346131    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"]
Oct 30 01:12:46.355428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.355394    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-hlhww"]
Oct 30 01:12:46.361706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.361671    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"]
Oct 30 01:12:46.371685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.371657    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-b8tbg"]
Oct 30 01:12:46.624683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.624601    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" path="/var/lib/kubelet/pods/17386f66-3f21-4f9e-8a12-9700848daaf9/volumes"
Oct 30 01:12:46.626893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.626868    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" path="/var/lib/kubelet/pods/1dfedb57-fd1c-4d6f-9d27-249f3a7f244a/volumes"
Oct 30 01:12:46.627821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.627802    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" path="/var/lib/kubelet/pods/6aa02a4f-e4f4-4b34-b165-e306f42c7c2a/volumes"
Oct 30 01:12:46.628612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:46.628596    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" path="/var/lib/kubelet/pods/bd9114fe-c972-4605-afa7-3975a55d3712/volumes"
Oct 30 01:12:47.081164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:47.080886    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"]
Oct 30 01:12:47.081164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:47.080949    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4294283d-c588f"]
Oct 30 01:12:47.115570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:47.115515    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" event={"ID":"7036635b-a1e2-4679-99dc-0e054dbbc2c5","Type":"ContainerStarted","Data":"2aa8b19e18604d8e040a7b935d179eb349d551e210c9060eeda28e7e6b15c4d7"}
Oct 30 01:12:47.115798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:47.115591    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" event={"ID":"7036635b-a1e2-4679-99dc-0e054dbbc2c5","Type":"ContainerStarted","Data":"64965dc7d9f7fc4516e9397d5b9a9b957ff1dbcf536012906a12c15251839ef2"}
Oct 30 01:12:47.116856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:47.116831    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:12:47.121928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:47.121726    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n" event={"ID":"afe71f7b-8700-4284-b72d-1af3a263cd4c","Type":"ContainerStarted","Data":"f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657"}
Oct 30 01:12:47.154189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:47.154134    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" podStartSLOduration=5.154114445 podStartE2EDuration="5.154114445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:47.153242141 +0000 UTC m=+1808.700932637" watchObservedRunningTime="2026-10-30 01:12:47.154114445 +0000 UTC m=+1808.701804940"
Oct 30 01:12:48.624852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:48.624818    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25c364f-5c34-4e57-8352-d63c08d49931" path="/var/lib/kubelet/pods/c25c364f-5c34-4e57-8352-d63c08d49931/volumes"
Oct 30 01:12:51.580729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:51.580688    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" podUID="a8f0a3cf-9833-409e-a859-de471cc262fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:53.184867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:53.184835    5104 generic.go:356] "Generic (PLEG): container finished" podID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerID="f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767" exitCode=0
Oct 30 01:12:53.185334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:53.184905    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" event={"ID":"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a","Type":"ContainerDied","Data":"f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767"}
Oct 30 01:12:53.185525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:53.185506    5104 scope.go:122] "RemoveContainer" containerID="f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767"
Oct 30 01:12:53.187744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:53.187714    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:53.188030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:53.187788    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:53.188030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:53.187838    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:12:54.215027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:54.214993    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" event={"ID":"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a","Type":"ContainerStarted","Data":"80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29"}
Oct 30 01:12:55.039144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.037403    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:12:55.040771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.040494    5104 patch_prober.go:29] interesting pod/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 01:12:55.040771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.040556    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 01:12:55.139738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.139709    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:12:55.233969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.233900    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f0a3cf-9833-409e-a859-de471cc262fb-kube-api-access-mk4l8\" (UniqueName: \"kubernetes.io/projected/a8f0a3cf-9833-409e-a859-de471cc262fb-kube-api-access-mk4l8\") pod \"a8f0a3cf-9833-409e-a859-de471cc262fb\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") "
Oct 30 01:12:55.234638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.234142    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f0a3cf-9833-409e-a859-de471cc262fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f0a3cf-9833-409e-a859-de471cc262fb-plugins-config-volume\") pod \"a8f0a3cf-9833-409e-a859-de471cc262fb\" (UID: \"a8f0a3cf-9833-409e-a859-de471cc262fb\") "
Oct 30 01:12:55.235313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.235272    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f0a3cf-9833-409e-a859-de471cc262fb-plugins-config-volume" pod "a8f0a3cf-9833-409e-a859-de471cc262fb" (UID: "a8f0a3cf-9833-409e-a859-de471cc262fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:55.237618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.237593    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f0a3cf-9833-409e-a859-de471cc262fb-kube-api-access-mk4l8" pod "a8f0a3cf-9833-409e-a859-de471cc262fb" (UID: "a8f0a3cf-9833-409e-a859-de471cc262fb"). InnerVolumeSpecName "kube-api-access-mk4l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:55.240351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.240327    5104 generic.go:356] "Generic (PLEG): container finished" podID="a8f0a3cf-9833-409e-a859-de471cc262fb" containerID="44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674" exitCode=0
Oct 30 01:12:55.240515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.240430    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"
Oct 30 01:12:55.242596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.241477    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" event={"ID":"a8f0a3cf-9833-409e-a859-de471cc262fb","Type":"ContainerDied","Data":"44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674"}
Oct 30 01:12:55.242596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.241540    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr" event={"ID":"a8f0a3cf-9833-409e-a859-de471cc262fb","Type":"ContainerDied","Data":"5e935def89b00add9dec4d9db78085894ef6c38d498efc925db1ea18d724bf8c"}
Oct 30 01:12:55.242596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.241573    5104 scope.go:122] "RemoveContainer" containerID="44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674"
Oct 30 01:12:55.264524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.264496    5104 scope.go:122] "RemoveContainer" containerID="44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674"
Oct 30 01:12:55.264996 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:12:55.264946    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674\": container with ID starting with 44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674 not found: ID does not exist" containerID="44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674"
Oct 30 01:12:55.265084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.264985    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674"} err="failed to get container status \"44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674\": rpc error: code = NotFound desc = could not find container \"44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674\": container with ID starting with 44f63852c80c26e0e21262f370ec17091b1996912af484c10326e93c6dd1d674 not found: ID does not exist"
Oct 30 01:12:55.321012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.320915    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"]
Oct 30 01:12:55.328764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.328727    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e233f-6489d58d4c-2d6lr"]
Oct 30 01:12:55.337102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.337074    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4l8\" (UniqueName: \"kubernetes.io/projected/a8f0a3cf-9833-409e-a859-de471cc262fb-kube-api-access-mk4l8\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.337102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.337099    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f0a3cf-9833-409e-a859-de471cc262fb-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.966939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.966902    5104 patch_prober.go:29] interesting pod/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 01:12:55.967144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:55.966968    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 01:12:56.253978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:56.253880    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:12:56.609922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:56.609835    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f0a3cf-9833-409e-a859-de471cc262fb" path="/var/lib/kubelet/pods/a8f0a3cf-9833-409e-a859-de471cc262fb/volumes"
Oct 30 01:12:56.967481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:56.966884    5104 patch_prober.go:29] interesting pod/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 01:12:56.967481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:56.966949    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 01:12:57.966854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:57.966818    5104 patch_prober.go:29] interesting pod/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 01:12:57.967272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:57.966885    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 01:12:58.140585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:58.140554    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:12:58.966075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:58.966039    5104 patch_prober.go:29] interesting pod/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 01:12:58.966232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:58.966114    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 01:12:59.152481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:12:59.150577    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:13:00.021377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:00.021346    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:13:00.023438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:00.023407    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:13:06.950481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:06.948733    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:13:08.156510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:08.155621    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"]
Oct 30 01:13:08.173793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:08.173397    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e233f-55dd9"]
Oct 30 01:13:08.580796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:08.580761    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f07832-2dfc-4a31-9677-564beb843a44" path="/var/lib/kubelet/pods/b4f07832-2dfc-4a31-9677-564beb843a44/volumes"
Oct 30 01:13:09.405480 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:13:09.399732    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:13:15.897836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:15.897744    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"]
Oct 30 01:13:15.899843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:15.899513    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy" containerID="cri-o://8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e" gracePeriod=30
Oct 30 01:13:20.696633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:20.696599    5104 generic.go:356] "Generic (PLEG): container finished" podID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerID="fedc832c0aa6907b55353184d1a82bfb260f71c7c32e406aa568c347167d10e7" exitCode=0
Oct 30 01:13:20.697219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:20.696679    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" event={"ID":"2b4d865c-5038-4871-8e4f-12cac274ca02","Type":"ContainerDied","Data":"fedc832c0aa6907b55353184d1a82bfb260f71c7c32e406aa568c347167d10e7"}
Oct 30 01:13:20.697299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:20.697280    5104 scope.go:122] "RemoveContainer" containerID="fedc832c0aa6907b55353184d1a82bfb260f71c7c32e406aa568c347167d10e7"
Oct 30 01:13:20.699523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:20.699487    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:20.699685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:20.699572    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:20.699685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:20.699615    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:21.356642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.356609    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:13:21.542982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.542946    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-kube-api-access-zw4bw\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-kube-api-access-zw4bw\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543024    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-certs\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543091    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-data\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543239    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1388f407-8532-47c6-b310-57e2c3302a07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1388f407-8532-47c6-b310-57e2c3302a07-istiod-ca-cert\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543267    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-credential-socket\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543297    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-socket\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543363    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-certs" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:21.543409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543385    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-data" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:21.543409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543401    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-envoy\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543439    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1388f407-8532-47c6-b310-57e2c3302a07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1388f407-8532-47c6-b310-57e2c3302a07-istio-podinfo\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543534    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-istio-token\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-istio-token\") pod \"1388f407-8532-47c6-b310-57e2c3302a07\" (UID: \"1388f407-8532-47c6-b310-57e2c3302a07\") "
Oct 30 01:13:21.543739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543589    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-credential-socket" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:21.543739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.543668    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-socket" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:21.544072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.544045    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1388f407-8532-47c6-b310-57e2c3302a07-istiod-ca-cert" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:21.544310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.544291    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.544310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.544310    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.544448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.544319    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1388f407-8532-47c6-b310-57e2c3302a07-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.544448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.544328    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.544448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.544337    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.545336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.545309    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-kube-api-access-zw4bw" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "kube-api-access-zw4bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:21.545594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.545573    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-envoy" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:21.545983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.545956    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1388f407-8532-47c6-b310-57e2c3302a07-istio-podinfo" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:21.546082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.545984    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-istio-token" pod "1388f407-8532-47c6-b310-57e2c3302a07" (UID: "1388f407-8532-47c6-b310-57e2c3302a07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:21.645422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.645386    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1388f407-8532-47c6-b310-57e2c3302a07-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.645422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.645419    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1388f407-8532-47c6-b310-57e2c3302a07-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.645422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.645428    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.645709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.645440    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw4bw\" (UniqueName: \"kubernetes.io/projected/1388f407-8532-47c6-b310-57e2c3302a07-kube-api-access-zw4bw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:21.716283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.716253    5104 generic.go:356] "Generic (PLEG): container finished" podID="1388f407-8532-47c6-b310-57e2c3302a07" containerID="8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e" exitCode=0
Oct 30 01:13:21.716794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.716523    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"
Oct 30 01:13:21.716794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.716759    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" event={"ID":"1388f407-8532-47c6-b310-57e2c3302a07","Type":"ContainerDied","Data":"8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e"}
Oct 30 01:13:21.716926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.716845    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr" event={"ID":"1388f407-8532-47c6-b310-57e2c3302a07","Type":"ContainerDied","Data":"e9bdb477faaeebec4a7d35595c62f5037c7f630e9acf31eca4466a4b30bd0835"}
Oct 30 01:13:21.716926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.716878    5104 scope.go:122] "RemoveContainer" containerID="8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e"
Oct 30 01:13:21.726137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.726104    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" event={"ID":"2b4d865c-5038-4871-8e4f-12cac274ca02","Type":"ContainerStarted","Data":"2e4ff9b021a141cc9ce2db8b50ea1d14f13ac0770b35b481731631dad538e689"}
Oct 30 01:13:21.732388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.732371    5104 scope.go:122] "RemoveContainer" containerID="d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91"
Oct 30 01:13:21.743411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.743392    5104 scope.go:122] "RemoveContainer" containerID="8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e"
Oct 30 01:13:21.744037 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:13:21.744015    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e\": container with ID starting with 8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e not found: ID does not exist" containerID="8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e"
Oct 30 01:13:21.744119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.744044    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e"} err="failed to get container status \"8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e\": rpc error: code = NotFound desc = could not find container \"8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e\": container with ID starting with 8af505ba0730434a3ccb959d7319f4d75bd6e2ee3f99012786ae83db9646397e not found: ID does not exist"
Oct 30 01:13:21.744119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.744063    5104 scope.go:122] "RemoveContainer" containerID="d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91"
Oct 30 01:13:21.744319 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:13:21.744300    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91\": container with ID starting with d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91 not found: ID does not exist" containerID="d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91"
Oct 30 01:13:21.744361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.744323    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91"} err="failed to get container status \"d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91\": rpc error: code = NotFound desc = could not find container \"d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91\": container with ID starting with d55c40d0eb712563f0ed7ee4b31c6259e926912e9cf4439ce639e4bae56a5f91 not found: ID does not exist"
Oct 30 01:13:21.826046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.825956    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"]
Oct 30 01:13:21.830394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.830370    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-xpchr"]
Oct 30 01:13:21.970976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.970947    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:13:21.971148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.971123    5104 patch_prober.go:29] interesting pod/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 01:13:21.971198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:21.971165    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 01:13:22.620874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:22.620832    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1388f407-8532-47c6-b310-57e2c3302a07" path="/var/lib/kubelet/pods/1388f407-8532-47c6-b310-57e2c3302a07/volumes"
Oct 30 01:13:22.977481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:22.976081    5104 patch_prober.go:29] interesting pod/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 01:13:22.977481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:22.976151    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 01:13:23.971652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:23.971618    5104 patch_prober.go:29] interesting pod/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 01:13:23.971842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:23.971672    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 01:13:24.971429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:24.971396    5104 patch_prober.go:29] interesting pod/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 01:13:24.971857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:24.971447    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 01:13:25.970941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:25.970904    5104 patch_prober.go:29] interesting pod/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 01:13:25.971131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:25.970976    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 01:13:26.971511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:26.971443    5104 patch_prober.go:29] interesting pod/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 01:13:26.971926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:26.971543    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 01:13:27.976480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:27.976432    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:13:27.976959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:27.976940    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:13:27.977766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:27.977744    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:13:29.371865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.371826    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"]
Oct 30 01:13:29.372312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.372045    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h" podUID="57324b74-cc8c-4b87-8bb7-fa51af899929" containerName="discovery" containerID="cri-o://ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f" gracePeriod=30
Oct 30 01:13:29.722094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.722060    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:13:29.813535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.813495    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-kubeconfig\") pod \"57324b74-cc8c-4b87-8bb7-fa51af899929\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") "
Oct 30 01:13:29.813750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.813578    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-token\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-token\") pod \"57324b74-cc8c-4b87-8bb7-fa51af899929\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") "
Oct 30 01:13:29.813750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.813649    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-dns-cert\") pod \"57324b74-cc8c-4b87-8bb7-fa51af899929\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") "
Oct 30 01:13:29.813750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.813709    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57324b74-cc8c-4b87-8bb7-fa51af899929-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57324b74-cc8c-4b87-8bb7-fa51af899929-local-certs\") pod \"57324b74-cc8c-4b87-8bb7-fa51af899929\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") "
Oct 30 01:13:29.813933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.813890    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-kube-api-access-xb5vs\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-kube-api-access-xb5vs\") pod \"57324b74-cc8c-4b87-8bb7-fa51af899929\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") "
Oct 30 01:13:29.814292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.813986    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-ca-configmap\") pod \"57324b74-cc8c-4b87-8bb7-fa51af899929\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") "
Oct 30 01:13:29.814292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.814092    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-cacerts\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-cacerts\") pod \"57324b74-cc8c-4b87-8bb7-fa51af899929\" (UID: \"57324b74-cc8c-4b87-8bb7-fa51af899929\") "
Oct 30 01:13:29.816617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.816567    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-ca-configmap" pod "57324b74-cc8c-4b87-8bb7-fa51af899929" (UID: "57324b74-cc8c-4b87-8bb7-fa51af899929"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:29.818694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.818650    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-kubeconfig" pod "57324b74-cc8c-4b87-8bb7-fa51af899929" (UID: "57324b74-cc8c-4b87-8bb7-fa51af899929"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:29.825513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.820894    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57324b74-cc8c-4b87-8bb7-fa51af899929-local-certs" pod "57324b74-cc8c-4b87-8bb7-fa51af899929" (UID: "57324b74-cc8c-4b87-8bb7-fa51af899929"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:29.825878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.825830    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-kube-api-access-xb5vs" pod "57324b74-cc8c-4b87-8bb7-fa51af899929" (UID: "57324b74-cc8c-4b87-8bb7-fa51af899929"). InnerVolumeSpecName "kube-api-access-xb5vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:29.825995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.825961    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-dns-cert" pod "57324b74-cc8c-4b87-8bb7-fa51af899929" (UID: "57324b74-cc8c-4b87-8bb7-fa51af899929"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:29.826072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.826000    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-token" pod "57324b74-cc8c-4b87-8bb7-fa51af899929" (UID: "57324b74-cc8c-4b87-8bb7-fa51af899929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:29.826843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.826819    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-cacerts" pod "57324b74-cc8c-4b87-8bb7-fa51af899929" (UID: "57324b74-cc8c-4b87-8bb7-fa51af899929"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:29.921480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.920719    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:29.921480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.920823    5104 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-dns-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:29.921480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.920842    5104 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57324b74-cc8c-4b87-8bb7-fa51af899929-local-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:29.921480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.920856    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5vs\" (UniqueName: \"kubernetes.io/projected/57324b74-cc8c-4b87-8bb7-fa51af899929-kube-api-access-xb5vs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:29.921480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.920871    5104 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-csr-ca-configmap\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:29.921480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.921001    5104 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-cacerts\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:29.921480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.921015    5104 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57324b74-cc8c-4b87-8bb7-fa51af899929-istio-kubeconfig\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:13:30.009591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.996341    5104 generic.go:356] "Generic (PLEG): container finished" podID="57324b74-cc8c-4b87-8bb7-fa51af899929" containerID="ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f" exitCode=0
Oct 30 01:13:30.009591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:29.997727    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"
Oct 30 01:13:30.029475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.027327    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h" event={"ID":"57324b74-cc8c-4b87-8bb7-fa51af899929","Type":"ContainerDied","Data":"ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f"}
Oct 30 01:13:30.029475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.027408    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h" event={"ID":"57324b74-cc8c-4b87-8bb7-fa51af899929","Type":"ContainerDied","Data":"8deee88d3a3508ea00132df7fd3f2ed24c1a0372a8f2d87519aa092564d0cc8e"}
Oct 30 01:13:30.029475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.027436    5104 scope.go:122] "RemoveContainer" containerID="ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f"
Oct 30 01:13:30.207536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.195396    5104 scope.go:122] "RemoveContainer" containerID="ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f"
Oct 30 01:13:30.207536 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:13:30.195905    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f\": container with ID starting with ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f not found: ID does not exist" containerID="ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f"
Oct 30 01:13:30.207536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.195948    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f"} err="failed to get container status \"ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f\": rpc error: code = NotFound desc = could not find container \"ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f\": container with ID starting with ba385eb6eb4299d8555caa3ac35cd248e3b7d78d0de4ed87101debc69978c46f not found: ID does not exist"
Oct 30 01:13:30.218480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.213659    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"]
Oct 30 01:13:30.227902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.223479    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6c2h"]
Oct 30 01:13:30.583078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:30.583039    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57324b74-cc8c-4b87-8bb7-fa51af899929" path="/var/lib/kubelet/pods/57324b74-cc8c-4b87-8bb7-fa51af899929/volumes"
Oct 30 01:13:32.015018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.014979    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"]
Oct 30 01:13:32.016309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016284    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016310    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016337    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016345    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016365    5104 cpu_manager.go:414] "containerMap: removing container" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016373    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016414    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016422    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy"
Oct 30 01:13:32.016437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016437    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a8f0a3cf-9833-409e-a859-de471cc262fb" containerName="payload-pre-processing"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016445    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f0a3cf-9833-409e-a859-de471cc262fb" containerName="payload-pre-processing"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016471    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016479    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016515    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016523    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016544    5104 cpu_manager.go:414] "containerMap: removing container" podUID="57324b74-cc8c-4b87-8bb7-fa51af899929" containerName="discovery"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016551    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57324b74-cc8c-4b87-8bb7-fa51af899929" containerName="discovery"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016566    5104 cpu_manager.go:414] "containerMap: removing container" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016573    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016733    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa02a4f-e4f4-4b34-b165-e306f42c7c2a" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016746    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="57324b74-cc8c-4b87-8bb7-fa51af899929" containerName="discovery"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016756    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9114fe-c972-4605-afa7-3975a55d3712" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016768    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016781    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1388f407-8532-47c6-b310-57e2c3302a07" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016794    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016809    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016821    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f0a3cf-9833-409e-a859-de471cc262fb" containerName="payload-pre-processing"
Oct 30 01:13:32.016894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.016831    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfedb57-fd1c-4d6f-9d27-249f3a7f244a" containerName="istio-proxy"
Oct 30 01:13:32.023078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.023055    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.051630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.051590    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"]
Oct 30 01:13:32.069050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.069018    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0e5711-openshift-default-dockercfg-d87hb\""
Oct 30 01:13:32.098175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098135    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkqq5\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-kube-api-access-rkqq5\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098268    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098338    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098407    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098514    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3716857a-a939-45b8-a24e-8f30afbc35ba-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098592    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098706    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098786    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.098893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.098840    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200231    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkqq5\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-kube-api-access-rkqq5\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200315    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200373    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200410    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200450    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3716857a-a939-45b8-a24e-8f30afbc35ba-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200545    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200653    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200697    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200765    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.200823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.200795    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.201196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.201166    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.201403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.201383    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.201522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.201500    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3716857a-a939-45b8-a24e-8f30afbc35ba-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.201765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.201745    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.202960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.202933    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.203868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.203844    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.233436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.227781    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkqq5\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-kube-api-access-rkqq5\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.237324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.237251    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.363539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.361356    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:32.890187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.890155    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"]
Oct 30 01:13:32.898119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.898076    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:32.898247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.898172    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:32.898247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:32.898212    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:33.035851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:33.035802    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" event={"ID":"3716857a-a939-45b8-a24e-8f30afbc35ba","Type":"ContainerStarted","Data":"c2ed956a6225d43ddff435e465f46c8a9514e4583f58c44453833388050ed3a0"}
Oct 30 01:13:33.036258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:33.035881    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" event={"ID":"3716857a-a939-45b8-a24e-8f30afbc35ba","Type":"ContainerStarted","Data":"febc991364315b9242ce84cc1ace982b62c2369e3e8008bd95c581c4d4518158"}
Oct 30 01:13:33.084564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:33.078111    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podStartSLOduration=2.078092084 podStartE2EDuration="2.078092084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:33.074966563 +0000 UTC m=+1854.622657060" watchObservedRunningTime="2026-10-30 01:13:33.078092084 +0000 UTC m=+1854.625782582"
Oct 30 01:13:33.361746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:33.361706    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:33.364181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:33.364153    5104 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 30 01:13:33.364265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:33.364212    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 30 01:13:34.368321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:34.368286    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:35.107914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:35.107883    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:35.109038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:35.109007    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:13:36.206190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.206091    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"]
Oct 30 01:13:36.207878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.207555    5104 cpu_manager.go:414] "containerMap: removing container" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy"
Oct 30 01:13:36.207878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.207585    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy"
Oct 30 01:13:36.208104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.207889    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="17386f66-3f21-4f9e-8a12-9700848daaf9" containerName="istio-proxy"
Oct 30 01:13:36.217124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.216742    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.228655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.228168    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"]
Oct 30 01:13:36.232914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.232877    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"openshift-service-ca.crt\""
Oct 30 01:13:36.271883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.271842    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"default-dockercfg-tfxjr\""
Oct 30 01:13:36.272067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.271920    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"auth-test-model-eacea3-kserve-self-signed-certs\""
Oct 30 01:13:36.331925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.331890    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"kube-root-ca.crt\""
Oct 30 01:13:36.339377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.339338    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-dshm\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.339580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.339426    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c84a28-d678-4a81-a8f5-0ecf73662359-tls-certs\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.339580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.339564    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5wf\" (UniqueName: \"kubernetes.io/projected/b9c84a28-d678-4a81-a8f5-0ecf73662359-kube-api-access-hl5wf\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.339692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.339672    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-model-cache\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.339739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.339733    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-tmp-dir\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.339901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.339880    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-home\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.442783    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-home\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.442993    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-dshm\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.443057    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c84a28-d678-4a81-a8f5-0ecf73662359-tls-certs\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.443094    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5wf\" (UniqueName: \"kubernetes.io/projected/b9c84a28-d678-4a81-a8f5-0ecf73662359-kube-api-access-hl5wf\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.443189    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-model-cache\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.443238    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-tmp-dir\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.443788    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-tmp-dir\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.444046    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-home\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.454476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.446994    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-model-cache\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.480528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.480187    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c84a28-d678-4a81-a8f5-0ecf73662359-tls-certs\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.484480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.480927    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-dshm\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.497483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.489519    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5wf\" (UniqueName: \"kubernetes.io/projected/b9c84a28-d678-4a81-a8f5-0ecf73662359-kube-api-access-hl5wf\") pod \"auth-test-model-eacea3-kserve-669498dd5-rxqjk\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.552485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.550943    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:36.938931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.938898    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"]
Oct 30 01:13:36.945966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.945936    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:36.955964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.955930    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"]
Oct 30 01:13:36.960680 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:13:36.960657    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9c84a28_d678_4a81_a8f5_0ecf73662359.slice/crio-c954f31e0a4e5071f092702b280e7d0473bc4b71fd57d36ae8221bb6866c3d78 WatchSource:0}: Error finding container c954f31e0a4e5071f092702b280e7d0473bc4b71fd57d36ae8221bb6866c3d78: Status 404 returned error can't find the container with id c954f31e0a4e5071f092702b280e7d0473bc4b71fd57d36ae8221bb6866c3d78
Oct 30 01:13:36.980873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:36.980834    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"]
Oct 30 01:13:37.016016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.015987    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52f1cb-openshift-default-dockercfg-bbr78\""
Oct 30 01:13:37.064392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-envoy\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.064600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064404    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istiod-ca-cert\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.064600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064440    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-data\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.064600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064522    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-token\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.064600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064581    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.064817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064638    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-credential-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.064817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064792    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5tc\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-kube-api-access-zz5tc\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.064943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.064925    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-podinfo\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.065070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.065034    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-certs\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.164133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.164093    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" event={"ID":"b9c84a28-d678-4a81-a8f5-0ecf73662359","Type":"ContainerStarted","Data":"3c66a64312fe5aac1754bcb9db178c87006678a90cb6ec1324e057136d0509b1"}
Oct 30 01:13:37.164133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.164143    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" event={"ID":"b9c84a28-d678-4a81-a8f5-0ecf73662359","Type":"ContainerStarted","Data":"c954f31e0a4e5071f092702b280e7d0473bc4b71fd57d36ae8221bb6866c3d78"}
Oct 30 01:13:37.166319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166290    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5tc\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-kube-api-access-zz5tc\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166376    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-podinfo\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166401    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-certs\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166447    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-envoy\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166492    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istiod-ca-cert\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166651    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-data\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166746    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-token\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166806    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.166893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166862    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-credential-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.167046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166897    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-certs\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.167046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.166998    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-data\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.167214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.167192    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-credential-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.167284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.167268    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istiod-ca-cert\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.167347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.167315    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.169088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.169066    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-envoy\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.169529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.169507    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-podinfo\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.187786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.187691    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5tc\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-kube-api-access-zz5tc\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.191715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.191650    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-token\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.207728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.207656    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" podStartSLOduration=1.207637265 podStartE2EDuration="1.207637265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:37.202592932 +0000 UTC m=+1858.750283428" watchObservedRunningTime="2026-10-30 01:13:37.207637265 +0000 UTC m=+1858.755327761"
Oct 30 01:13:37.264057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.264018    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:37.552468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.552410    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:37.853571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.830520    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"]
Oct 30 01:13:37.894353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.893077    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:37.894353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.893192    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:37.894353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:37.893272    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:13:38.235479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.229899    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" event={"ID":"3fc8515d-c0ef-4f1f-9c23-536f9340fb74","Type":"ContainerStarted","Data":"8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60"}
Oct 30 01:13:38.235479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.229981    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" event={"ID":"3fc8515d-c0ef-4f1f-9c23-536f9340fb74","Type":"ContainerStarted","Data":"099aa262187982a58b55e89c07647fd2a95ce88d0e329ac9dd7246cb4bc1ec2c"}
Oct 30 01:13:38.272712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.267644    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:38.809652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.809595    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podStartSLOduration=2.80956781 podStartE2EDuration="2.80956781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:38.292751759 +0000 UTC m=+1859.840442256" watchObservedRunningTime="2026-10-30 01:13:38.80956781 +0000 UTC m=+1860.357258309"
Oct 30 01:13:38.813164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.810865    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"]
Oct 30 01:13:38.845443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.845411    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"]
Oct 30 01:13:38.845654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.845561    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:38.877725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.870866    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0e5711\""
Oct 30 01:13:38.877725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.871076    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0e5711-dockercfg-tv2d5\""
Oct 30 01:13:38.981628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:38.980610    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"]
Oct 30 01:13:39.232615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.115859    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pcp\" (UniqueName: \"kubernetes.io/projected/f7636dbe-33ba-45d3-a6a8-884db1a2c880-kube-api-access-c7pcp\") pod \"payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:39.232615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.116070    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7636dbe-33ba-45d3-a6a8-884db1a2c880-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:39.232615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.224213    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pcp\" (UniqueName: \"kubernetes.io/projected/f7636dbe-33ba-45d3-a6a8-884db1a2c880-kube-api-access-c7pcp\") pod \"payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:39.232615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.224450    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7636dbe-33ba-45d3-a6a8-884db1a2c880-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:39.232615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.231756    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7636dbe-33ba-45d3-a6a8-884db1a2c880-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:39.253049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.252769    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:39.318611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.318579    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"]
Oct 30 01:13:39.324740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.324709    5104 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 01:13:39.325040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.325014    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 01:13:39.342372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.342080    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwc62\" (UniqueName: \"kubernetes.io/projected/ca181706-ab97-4fec-bc71-6b55126060f3-kube-api-access-zwc62\") pod \"payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:39.342372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.342146    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca181706-ab97-4fec-bc71-6b55126060f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:39.380608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.380573    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pcp\" (UniqueName: \"kubernetes.io/projected/f7636dbe-33ba-45d3-a6a8-884db1a2c880-kube-api-access-c7pcp\") pod \"payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:39.457364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.456753    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc62\" (UniqueName: \"kubernetes.io/projected/ca181706-ab97-4fec-bc71-6b55126060f3-kube-api-access-zwc62\") pod \"payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:39.457364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.457287    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca181706-ab97-4fec-bc71-6b55126060f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:39.458379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.458347    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca181706-ab97-4fec-bc71-6b55126060f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:39.460192 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:13:39.460145    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:13:39.527838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.527659    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwc62\" (UniqueName: \"kubernetes.io/projected/ca181706-ab97-4fec-bc71-6b55126060f3-kube-api-access-zwc62\") pod \"payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:39.567039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.566994    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:39.733829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:39.732903    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:40.011052 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:13:40.011017    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7636dbe_33ba_45d3_a6a8_884db1a2c880.slice/crio-4bbe42fcdd3c77f4cb972c4f2fd0e66792be81863c9c2c08fa02b70602c4810d WatchSource:0}: Error finding container 4bbe42fcdd3c77f4cb972c4f2fd0e66792be81863c9c2c08fa02b70602c4810d: Status 404 returned error can't find the container with id 4bbe42fcdd3c77f4cb972c4f2fd0e66792be81863c9c2c08fa02b70602c4810d
Oct 30 01:13:40.012294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.012261    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"]
Oct 30 01:13:40.121544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.113954    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"]
Oct 30 01:13:40.269022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.268618    5104 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 01:13:40.269022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.268672    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 01:13:40.338748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.338718    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" event={"ID":"ca181706-ab97-4fec-bc71-6b55126060f3","Type":"ContainerStarted","Data":"a04158a53114cdecd0e3a2114f50e0e972e6c3f9f15ba64577b7b32b81256ad4"}
Oct 30 01:13:40.340770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.340742    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" event={"ID":"f7636dbe-33ba-45d3-a6a8-884db1a2c880","Type":"ContainerStarted","Data":"6b7f6afa7887eba8c3cabb4e85d1195ef4b73f54f8860d16fb48008116715d0b"}
Oct 30 01:13:40.340885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.340787    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" event={"ID":"f7636dbe-33ba-45d3-a6a8-884db1a2c880","Type":"ContainerStarted","Data":"4bbe42fcdd3c77f4cb972c4f2fd0e66792be81863c9c2c08fa02b70602c4810d"}
Oct 30 01:13:40.401740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:40.401668    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" podStartSLOduration=2.401642517 podStartE2EDuration="2.401642517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:40.391547695 +0000 UTC m=+1861.939238191" watchObservedRunningTime="2026-10-30 01:13:40.401642517 +0000 UTC m=+1861.949333016"
Oct 30 01:13:41.264745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:41.264705    5104 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 01:13:41.264944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:41.264760    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 01:13:41.351697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:41.351663    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" event={"ID":"ca181706-ab97-4fec-bc71-6b55126060f3","Type":"ContainerStarted","Data":"d64b4f7afd768ca9cb2f0e90196fe2ea81c2371e6b7c629c6a43b6ced6b1ade7"}
Oct 30 01:13:41.352218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:41.351948    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:41.352218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:41.351970    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:42.264980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:42.264942    5104 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 01:13:42.265252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:42.265009    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 01:13:43.264771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:43.264737    5104 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 01:13:43.265178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:43.264793    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 01:13:44.264364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:44.264331    5104 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 01:13:44.264585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:44.264384    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 01:13:45.283489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:45.272866    5104 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 01:13:45.283489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:45.272934    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 01:13:46.076821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.076783    5104 scope.go:122] "RemoveContainer" containerID="82875eed34f15d7bbc989b7f8ba7d893791258c62902f337bb7644c41eca8146"
Oct 30 01:13:46.093868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.093839    5104 scope.go:122] "RemoveContainer" containerID="605218bf9d18199485b8026066678980c8420d96fb676e77bcd8b9904c4ec134"
Oct 30 01:13:46.269743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.269709    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:46.270227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.270202    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:46.270992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.270970    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:13:46.324124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.324060    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" podStartSLOduration=8.324039296 podStartE2EDuration="8.324039296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:41.402875901 +0000 UTC m=+1862.950566398" watchObservedRunningTime="2026-10-30 01:13:46.324039296 +0000 UTC m=+1867.871729795"
Oct 30 01:13:46.552579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.552539    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:46.565883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:46.565856    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:47.467779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:47.467749    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:13:52.101570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.101451    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"]
Oct 30 01:13:52.113445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.111384    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.124601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.124567    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"]
Oct 30 01:13:52.134662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.134613    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4dde48\"/\"default-dockercfg-6rw5k\""
Oct 30 01:13:52.145415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.145384    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4dde48\"/\"openshift-service-ca.crt\""
Oct 30 01:13:52.145650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.145441    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4dde48\"/\"kube-root-ca.crt\""
Oct 30 01:13:52.145650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.145468    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4dde48\"/\"test-model-4dde48-kserve-self-signed-certs\""
Oct 30 01:13:52.234804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.234767    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-dshm\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.235011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.234971    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-model-cache\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.235123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.235106    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhhbd\" (UniqueName: \"kubernetes.io/projected/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-kube-api-access-nhhbd\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.235251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.235224    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tls-certs\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.235308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.235281    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-home\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.235467 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.235430    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tmp-dir\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.344485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.337914    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhhbd\" (UniqueName: \"kubernetes.io/projected/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-kube-api-access-nhhbd\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.344485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.338421    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tls-certs\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.344485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.338503    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-home\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.344485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.339018    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-home\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.364606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.354993    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tmp-dir\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.364606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.355406    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-dshm\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.364606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.356076    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-model-cache\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.364606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.362559    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tmp-dir\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.372546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.370873    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-model-cache\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.397490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.383656    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:13:52.397490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.391387    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tls-certs\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.397490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.397034    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhhbd\" (UniqueName: \"kubernetes.io/projected/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-kube-api-access-nhhbd\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.398114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.398089    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:13:52.431867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.431821    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-dshm\") pod \"test-model-4dde48-kserve-5f99d5bc77-cc294\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") " pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.472160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.471806    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:13:52.949377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:52.949347    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"]
Oct 30 01:13:52.949680 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:13:52.949656    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1d4415e_f0a4_4ce4_bfbb_4b63b590f9aa.slice/crio-b039c7535584533cd1e6ce4057ccb1c745c61f2b0c1cd01ae47a573c9fe0e464 WatchSource:0}: Error finding container b039c7535584533cd1e6ce4057ccb1c745c61f2b0c1cd01ae47a573c9fe0e464: Status 404 returned error can't find the container with id b039c7535584533cd1e6ce4057ccb1c745c61f2b0c1cd01ae47a573c9fe0e464
Oct 30 01:13:53.620680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:53.620639    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" event={"ID":"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa","Type":"ContainerStarted","Data":"59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f"}
Oct 30 01:13:53.621215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:53.621190    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" event={"ID":"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa","Type":"ContainerStarted","Data":"b039c7535584533cd1e6ce4057ccb1c745c61f2b0c1cd01ae47a573c9fe0e464"}
Oct 30 01:13:53.661056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:53.660992    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" podStartSLOduration=1.660974096 podStartE2EDuration="1.660974096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:53.658079017 +0000 UTC m=+1875.205769523" watchObservedRunningTime="2026-10-30 01:13:53.660974096 +0000 UTC m=+1875.208664594"
Oct 30 01:13:54.475774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:13:54.475742    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:14:02.475765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:02.475728    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:14:02.489167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:02.489136    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:14:02.717349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:02.717312    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:14:02.855158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:02.855078    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"]
Oct 30 01:14:02.860367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:02.860340    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:02.879878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:02.879843    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e92dab-openshift-default-dockercfg-9pm7p\""
Oct 30 01:14:02.898448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:02.898416    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"]
Oct 30 01:14:03.004705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.004665    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.004705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.004713    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqqx\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-kube-api-access-xqqqx\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.004960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.004867    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-credential-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.004960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.004939    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-podinfo\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.005101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.005076    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istiod-ca-cert\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.005143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.005125    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-data\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.005216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.005201    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-certs\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.005278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.005266    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-token\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.005381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.005357    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-envoy\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107167    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-credential-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107312    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-podinfo\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107383    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istiod-ca-cert\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107416    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-data\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107531    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-certs\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107795    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-data\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107819    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-token\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107868    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-certs\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107886    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-envoy\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107663    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-credential-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.107996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.107967    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.108181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.108005    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqqx\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-kube-api-access-xqqqx\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.108238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.108217    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istiod-ca-cert\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.108314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.108270    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.110608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.110573    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-envoy\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.110765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.110640    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-podinfo\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.126804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.126760    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-token\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.126985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.126856    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqqx\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-kube-api-access-xqqqx\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.196889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.196849    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:03.521201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.521168    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"]
Oct 30 01:14:03.526667 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:14:03.526605    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d47b9bd_4728_4846_80d9_ac2717fd0c74.slice/crio-c1622e2b3b5510f7c414d956a8194a967fb3dde585ef44986de75afc4f21321a WatchSource:0}: Error finding container c1622e2b3b5510f7c414d956a8194a967fb3dde585ef44986de75afc4f21321a: Status 404 returned error can't find the container with id c1622e2b3b5510f7c414d956a8194a967fb3dde585ef44986de75afc4f21321a
Oct 30 01:14:03.530672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.530630    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:14:03.530795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.530708    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:14:03.530795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.530740    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:14:03.720475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.720074    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" event={"ID":"2d47b9bd-4728-4846-80d9-ac2717fd0c74","Type":"ContainerStarted","Data":"51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9"}
Oct 30 01:14:03.720475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.720136    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" event={"ID":"2d47b9bd-4728-4846-80d9-ac2717fd0c74","Type":"ContainerStarted","Data":"c1622e2b3b5510f7c414d956a8194a967fb3dde585ef44986de75afc4f21321a"}
Oct 30 01:14:03.759807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:03.759739    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podStartSLOduration=1.759698485 podStartE2EDuration="1.759698485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:03.75853391 +0000 UTC m=+1885.306224406" watchObservedRunningTime="2026-10-30 01:14:03.759698485 +0000 UTC m=+1885.307388973"
Oct 30 01:14:04.197534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:04.197499    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:04.199441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:04.199414    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 01:14:04.199624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:04.199483    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 01:14:05.203977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:05.203932    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 01:14:05.204534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:05.204013    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 01:14:06.014712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.014674    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"]
Oct 30 01:14:06.021640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.021611    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.042721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.042658    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-34e896\"/\"kube-root-ca.crt\""
Oct 30 01:14:06.042911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.042775    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-34e896\"/\"openshift-service-ca.crt\""
Oct 30 01:14:06.044607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.044472    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"]
Oct 30 01:14:06.054177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.054151    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-34e896\"/\"test-model-34e896-kserve-self-signed-certs\""
Oct 30 01:14:06.054286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.054208    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-34e896\"/\"default-dockercfg-kvcg4\""
Oct 30 01:14:06.071428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.071391    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tmp-dir\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.071742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.071713    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-home\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.071895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.071793    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pqg8\" (UniqueName: \"kubernetes.io/projected/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-kube-api-access-8pqg8\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.071895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.071835    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tls-certs\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.071895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.071892    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-dshm\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.072086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.071922    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-model-cache\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.177570    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tmp-dir\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.177880    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-home\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.177957    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pqg8\" (UniqueName: \"kubernetes.io/projected/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-kube-api-access-8pqg8\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.177997    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tls-certs\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.178061    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-dshm\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.178090    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-model-cache\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.178650    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-model-cache\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.178919    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tmp-dir\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.188481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.179159    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-home\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.203512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.202843    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 01:14:06.203512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.202937    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 01:14:06.206202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.206173    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-dshm\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.206970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.206948    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pqg8\" (UniqueName: \"kubernetes.io/projected/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-kube-api-access-8pqg8\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.207936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.207914    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tls-certs\") pod \"test-model-34e896-kserve-859b999c49-dtw7g\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") " pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.352915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.352825    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:06.816578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:06.816544    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"]
Oct 30 01:14:07.199940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:07.199892    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 01:14:07.200272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:07.199961    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 01:14:07.804663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:07.804619    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" event={"ID":"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8","Type":"ContainerStarted","Data":"fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164"}
Oct 30 01:14:07.821781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:07.821739    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" event={"ID":"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8","Type":"ContainerStarted","Data":"766906a4107b41ec2d9167678ececaed2aa2359eae56a3e69951045172a33e31"}
Oct 30 01:14:07.868875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:07.868820    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" podStartSLOduration=2.8688028230000002 podStartE2EDuration="2.868802823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:07.864807199 +0000 UTC m=+1889.412497707" watchObservedRunningTime="2026-10-30 01:14:07.868802823 +0000 UTC m=+1889.416493319"
Oct 30 01:14:08.202534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:08.198841    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 01:14:08.202534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:08.198900    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 01:14:08.354338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:08.354303    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:09.200487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:09.197518    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 01:14:09.200487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:09.197573    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 01:14:09.338476 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:14:09.338389    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:14:10.197890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:10.197843    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 01:14:10.198080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:10.197923    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 01:14:11.203819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.203789    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:11.204377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.204352    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:11.205228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.205207    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:14:11.527983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.527895    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"]
Oct 30 01:14:11.536373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.536338    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.550769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.550549    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"]
Oct 30 01:14:11.552674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.552650    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"auth-test-model-69dc90-kserve-self-signed-certs\""
Oct 30 01:14:11.552922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.552900    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"kube-root-ca.crt\""
Oct 30 01:14:11.553052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.553007    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"openshift-service-ca.crt\""
Oct 30 01:14:11.553746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.553712    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"default-dockercfg-tx6rt\""
Oct 30 01:14:11.693087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.693049    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-dshm\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.693296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.693171    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tls-certs\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.693296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.693271    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tmp-dir\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.693432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.693337    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-model-cache\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.693538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.693443    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-home\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.693756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.693731    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzlv\" (UniqueName: \"kubernetes.io/projected/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-kube-api-access-rlzlv\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.804534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.804128    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-model-cache\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.804534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.804249    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-home\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.804534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.804411    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzlv\" (UniqueName: \"kubernetes.io/projected/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-kube-api-access-rlzlv\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.804847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.804599    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-dshm\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.804847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.804628    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tls-certs\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.804847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.804842    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tmp-dir\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.814478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.805380    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tmp-dir\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.814478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.805677    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-model-cache\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.814478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.805964    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-home\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.886225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.886190    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tls-certs\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.895893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.895848    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-dshm\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:11.990194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:11.971407    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzlv\" (UniqueName: \"kubernetes.io/projected/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-kube-api-access-rlzlv\") pod \"auth-test-model-69dc90-kserve-7c488c7557-vh79v\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:12.158716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:12.158644    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:12.472429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:12.472401    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"]
Oct 30 01:14:12.473573 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:14:12.473534    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44ad83d4_e9ba_43a8_9d9a_c69a94c7ad10.slice/crio-21b6e51b51ca49075ffd7b8e3d8bd2f4cfd3308760800eb316f27a43e9c0fca2 WatchSource:0}: Error finding container 21b6e51b51ca49075ffd7b8e3d8bd2f4cfd3308760800eb316f27a43e9c0fca2: Status 404 returned error can't find the container with id 21b6e51b51ca49075ffd7b8e3d8bd2f4cfd3308760800eb316f27a43e9c0fca2
Oct 30 01:14:12.952085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:12.952044    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" event={"ID":"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10","Type":"ContainerStarted","Data":"114ef00aa6842c8ce6cb71fbfb122236959c4fd1400682a7b31242e5220699f2"}
Oct 30 01:14:12.952301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:12.952097    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" event={"ID":"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10","Type":"ContainerStarted","Data":"21b6e51b51ca49075ffd7b8e3d8bd2f4cfd3308760800eb316f27a43e9c0fca2"}
Oct 30 01:14:12.992920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:12.992852    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" podStartSLOduration=1.9928353749999999 podStartE2EDuration="1.992835375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:12.988975352 +0000 UTC m=+1894.536665847" watchObservedRunningTime="2026-10-30 01:14:12.992835375 +0000 UTC m=+1894.540525872"
Oct 30 01:14:13.168031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:13.167986    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:16.354351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:16.354319    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:16.448232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:16.447877    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:17.078482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:17.073170    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:14:17.285762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:17.285730    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:14:19.711486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:19.711341    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" podUID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.236:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:19.865777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:19.865727    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:19.899613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:19.899563    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.057684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.057577    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.101749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.101693    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.490618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.490557    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.725396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.502607    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.232:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.725396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.562262    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.725396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.653567    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.725396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.712712    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" podUID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.236:8000/ready\": context deadline exceeded"
Oct 30 01:14:20.868538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.866936    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.907599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.901764    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:20.907599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.905016    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:14:20.907599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.905079    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:14:20.907599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.905112    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:14:20.907599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:20.905146    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:14:21.099480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:21.097548    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" podUID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.238:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:21.107979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:21.107932    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:14:21.119530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:21.111091    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:14:21.320372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:21.320336    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:14:21.541125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:21.540143    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:14:21.562718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:21.562608    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:14:22.158832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:22.158800    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:22.172032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:22.172004    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:22.377018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:22.376983    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:14:23.987638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:23.987601    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"]
Oct 30 01:14:23.997482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:23.996728    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.024559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.021948    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"default-dockercfg-qr8vs\""
Oct 30 01:14:24.029243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.029204    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"]
Oct 30 01:14:24.034727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.034697    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"kube-root-ca.crt\""
Oct 30 01:14:24.035268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.034718    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"auth-test-model-29145c-kserve-self-signed-certs\""
Oct 30 01:14:24.035268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.034938    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"openshift-service-ca.crt\""
Oct 30 01:14:24.075276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.075239    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tmp-dir\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.075552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.075340    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tls-certs\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.075552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.075392    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-kube-api-access-rxdlc\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.075552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.075473    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-model-cache\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.075552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.075494    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-home\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.075552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.075554    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-dshm\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.177574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.177529    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tmp-dir\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.177757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.177634    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tls-certs\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.177757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.177679    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-kube-api-access-rxdlc\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.177757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.177742    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-model-cache\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.177988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.177761    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-home\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.177988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.177915    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-dshm\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.178137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.178009    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tmp-dir\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.178137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.178057    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-model-cache\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.178225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.178154    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-home\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.180245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.180214    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-dshm\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.180609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.180584    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tls-certs\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.197572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.197535    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-kube-api-access-rxdlc\") pod \"auth-test-model-29145c-kserve-5986d669f7-mwkx5\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") " pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.360252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.359794    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:24.763329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:24.763233    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"]
Oct 30 01:14:25.461231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:25.461191    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5" event={"ID":"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4","Type":"ContainerStarted","Data":"6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56"}
Oct 30 01:14:25.461758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:25.461249    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5" event={"ID":"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4","Type":"ContainerStarted","Data":"c6fb1ef5a8e5b731c1a5b8423de4725d1716a3c72dbbb0b712dfa37cfac726c3"}
Oct 30 01:14:25.503200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:25.503133    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5" podStartSLOduration=2.503113274 podStartE2EDuration="2.503113274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:25.496858913 +0000 UTC m=+1907.044549410" watchObservedRunningTime="2026-10-30 01:14:25.503113274 +0000 UTC m=+1907.050803771"
Oct 30 01:14:26.361257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:26.361225    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:27.028468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:27.028423    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"]
Oct 30 01:14:27.039593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:27.039566    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5534addc-jqhjh"]
Oct 30 01:14:28.582309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:28.582268    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17bc456-2e63-42df-954a-88c26f991768" path="/var/lib/kubelet/pods/b17bc456-2e63-42df-954a-88c26f991768/volumes"
Oct 30 01:14:32.595670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.595634    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"]
Oct 30 01:14:32.602167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.602142    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.618403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.618373    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"]
Oct 30 01:14:32.623133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.623104    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-82e9bf\"/\"openshift-service-ca.crt\""
Oct 30 01:14:32.623419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.623303    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-82e9bf\"/\"kube-root-ca.crt\""
Oct 30 01:14:32.629364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.629334    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-82e9bf\"/\"default-dockercfg-kf8g6\""
Oct 30 01:14:32.629364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.629334    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-82e9bf\"/\"test-model-82e9bf-kserve-self-signed-certs\""
Oct 30 01:14:32.646105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.646071    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-tmp-dir\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.646422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.646404    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwdp\" (UniqueName: \"kubernetes.io/projected/fd630c4e-e0f2-4553-985f-94d87184f054-kube-api-access-dqwdp\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.646651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.646622    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-dshm\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.646795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.646689    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-model-cache\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.646935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.646906    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-home\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.647023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.647002    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd630c4e-e0f2-4553-985f-94d87184f054-tls-certs\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.749241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.749201    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-tmp-dir\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.749451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.749278    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwdp\" (UniqueName: \"kubernetes.io/projected/fd630c4e-e0f2-4553-985f-94d87184f054-kube-api-access-dqwdp\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.749451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.749385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-dshm\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.749451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.749441    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-model-cache\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.749709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.749567    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-home\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.749709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.749644    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd630c4e-e0f2-4553-985f-94d87184f054-tls-certs\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.749843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.749815    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-tmp-dir\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.750209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.750181    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-home\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.777871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.777831    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-model-cache\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.813481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.778950    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-dshm\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.813481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.808222    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwdp\" (UniqueName: \"kubernetes.io/projected/fd630c4e-e0f2-4553-985f-94d87184f054-kube-api-access-dqwdp\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.813481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.808842    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd630c4e-e0f2-4553-985f-94d87184f054-tls-certs\") pod \"test-model-82e9bf-kserve-68db7b8cf-q2n82\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") " pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:32.925479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:32.922800    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:33.372760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:33.372571    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"]
Oct 30 01:14:33.671807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:33.671775    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82" event={"ID":"fd630c4e-e0f2-4553-985f-94d87184f054","Type":"ContainerStarted","Data":"e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85"}
Oct 30 01:14:33.672213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:33.671820    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82" event={"ID":"fd630c4e-e0f2-4553-985f-94d87184f054","Type":"ContainerStarted","Data":"dd0d02807d66b5dfce56fd52c3d586968ec24c999a2e1bcd9139e1990fdfd788"}
Oct 30 01:14:33.716508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:33.711008    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82" podStartSLOduration=1.710989132 podStartE2EDuration="1.710989132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:33.707534068 +0000 UTC m=+1915.255224564" watchObservedRunningTime="2026-10-30 01:14:33.710989132 +0000 UTC m=+1915.258679628"
Oct 30 01:14:33.928539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:33.927695    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:34.360396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:34.360314    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:34.373743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:34.373708    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:34.695617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:34.695580    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:14:39.114492 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:14:39.114423    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:14:42.923487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:42.923442    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:42.936692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:42.936667    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:43.786804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:43.786768    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:14:44.732711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.732675    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"]
Oct 30 01:14:44.741366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.741335    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.760738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.758290    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"]
Oct 30 01:14:44.760738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.760562    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e69dee\"/\"openshift-service-ca.crt\""
Oct 30 01:14:44.760738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.760654    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e69dee\"/\"kube-root-ca.crt\""
Oct 30 01:14:44.795353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.795306    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5802f4e-a81e-4630-98ce-b308b35f457b-tls-certs\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.795572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.795404    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-model-cache\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.795572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.795505    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-dshm\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.795777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.795584    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/b5802f4e-a81e-4630-98ce-b308b35f457b-kube-api-access-dqn8b\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.795850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.795807    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-tmp-dir\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.795921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.795900    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-home\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.836697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.836662    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e69dee\"/\"default-dockercfg-wtf6b\""
Oct 30 01:14:44.836936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.836772    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e69dee\"/\"test-model-e69dee-kserve-self-signed-certs\""
Oct 30 01:14:44.898050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.898008    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-dshm\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.898340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.898134    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/b5802f4e-a81e-4630-98ce-b308b35f457b-kube-api-access-dqn8b\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.899135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.899110    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-tmp-dir\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.899793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.899643    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-tmp-dir\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.900028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.900012    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-home\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.900520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.900501    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5802f4e-a81e-4630-98ce-b308b35f457b-tls-certs\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.900732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.900718    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-model-cache\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.901226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.901208    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-model-cache\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.901353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.900361    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-home\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.904680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.904653    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5802f4e-a81e-4630-98ce-b308b35f457b-tls-certs\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.905083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.905062    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-dshm\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:44.929716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:44.929659    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/b5802f4e-a81e-4630-98ce-b308b35f457b-kube-api-access-dqn8b\") pod \"test-model-e69dee-kserve-8594c4554c-2l6r5\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") " pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:45.093540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:45.077990    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:45.558889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:45.558852    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"]
Oct 30 01:14:45.559377 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:14:45.559346    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5802f4e_a81e_4630_98ce_b308b35f457b.slice/crio-aa4c797a1d11d75d91491d85d3090e7fa9d36d955eca8d375b6b23bbe6cc98f7 WatchSource:0}: Error finding container aa4c797a1d11d75d91491d85d3090e7fa9d36d955eca8d375b6b23bbe6cc98f7: Status 404 returned error can't find the container with id aa4c797a1d11d75d91491d85d3090e7fa9d36d955eca8d375b6b23bbe6cc98f7
Oct 30 01:14:45.951486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:45.949620    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5" event={"ID":"b5802f4e-a81e-4630-98ce-b308b35f457b","Type":"ContainerStarted","Data":"2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210"}
Oct 30 01:14:45.951486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:45.949684    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5" event={"ID":"b5802f4e-a81e-4630-98ce-b308b35f457b","Type":"ContainerStarted","Data":"aa4c797a1d11d75d91491d85d3090e7fa9d36d955eca8d375b6b23bbe6cc98f7"}
Oct 30 01:14:45.997690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:45.997604    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5" podStartSLOduration=1.997582312 podStartE2EDuration="1.997582312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:45.990917647 +0000 UTC m=+1927.538608165" watchObservedRunningTime="2026-10-30 01:14:45.997582312 +0000 UTC m=+1927.545272809"
Oct 30 01:14:46.084709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:46.084670    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:46.203533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:46.202616    5104 scope.go:122] "RemoveContainer" containerID="6fa1c10c73c5f949bbff9676968c9e17abdf0e7e9333a662a99bd2495ad66779"
Oct 30 01:14:46.246725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:46.246119    5104 scope.go:122] "RemoveContainer" containerID="3c4a719377817066357c7ef1c1ff08ab2661dd2b54f1d967b703f38869760bce"
Oct 30 01:14:46.265676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:46.265612    5104 scope.go:122] "RemoveContainer" containerID="23f829229dd55c87d8c33831d15bf1194b59fa03b23faf890da3a513baf1b768"
Oct 30 01:14:49.454172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:49.453948    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.232:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:49.508802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:49.508711    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:49.630480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:49.629559    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:49.647940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:49.646168    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:49.662482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:49.651370    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:14:52.033991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:52.033959    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"]
Oct 30 01:14:52.046315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:52.046284    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d81291-rshdx"]
Oct 30 01:14:52.610725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:52.610686    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa09b2dd-244c-445d-9a55-ad46ce5b15e5" path="/var/lib/kubelet/pods/aa09b2dd-244c-445d-9a55-ad46ce5b15e5/volumes"
Oct 30 01:14:55.084815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:55.084722    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:55.102886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:55.102853    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:55.140905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:55.140872    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:14:59.175066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:59.175014    5104 generic.go:356] "Generic (PLEG): container finished" podID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerID="c2ed956a6225d43ddff435e465f46c8a9514e4583f58c44453833388050ed3a0" exitCode=0
Oct 30 01:14:59.175661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:59.175089    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" event={"ID":"3716857a-a939-45b8-a24e-8f30afbc35ba","Type":"ContainerDied","Data":"c2ed956a6225d43ddff435e465f46c8a9514e4583f58c44453833388050ed3a0"}
Oct 30 01:14:59.175757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:59.175670    5104 scope.go:122] "RemoveContainer" containerID="c2ed956a6225d43ddff435e465f46c8a9514e4583f58c44453833388050ed3a0"
Oct 30 01:14:59.177908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:59.177878    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:14:59.178047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:59.177945    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:14:59.178047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:14:59.177980    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:00.209680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.207351    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" event={"ID":"3716857a-a939-45b8-a24e-8f30afbc35ba","Type":"ContainerStarted","Data":"9af10a350511a482bf7c4d81c485965fd1be8c367bffdf4d20134e9a6fcb957c"}
Oct 30 01:15:00.224436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.224403    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"]
Oct 30 01:15:00.238619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.237745    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"]
Oct 30 01:15:00.242615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.242583    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.250241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.248920    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"]
Oct 30 01:15:00.250877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.250834    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.255218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.254949    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"]
Oct 30 01:15:00.255218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.255080    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.259498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.259148    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:15:00.264281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.264179    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:15:00.265871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.265850    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:15:00.266980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.266947    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg4cv\" (UniqueName: \"kubernetes.io/projected/aad6b4c4-517f-41a2-b986-e7367bef493b-kube-api-access-cg4cv\") pod \"maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.267115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.267093    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zd9\" (UniqueName: \"kubernetes.io/projected/ee09f6bd-6a43-4113-9d5c-84029ec53140-kube-api-access-h5zd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.267878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.267857    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4bx\" (UniqueName: \"kubernetes.io/projected/0489e1db-1b58-4955-966b-82a17015a4be-kube-api-access-nb4bx\") pod \"maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.268110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.268089    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee09f6bd-6a43-4113-9d5c-84029ec53140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.268551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.268531    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0489e1db-1b58-4955-966b-82a17015a4be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.268688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.268652    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"]
Oct 30 01:15:00.268881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.268856    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad6b4c4-517f-41a2-b986-e7367bef493b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.269125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.268642    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.292949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.292921    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.324042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.324000    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"]
Oct 30 01:15:00.347019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.338655    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"]
Oct 30 01:15:00.349417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.338886    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.363781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.363745    5104 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 30 01:15:00.364010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.363802    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 30 01:15:00.364730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.364702    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"]
Oct 30 01:15:00.365796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.364938    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.370671    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b270123-f6a5-4202-b8e3-699b84d55ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.370723    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4cv\" (UniqueName: \"kubernetes.io/projected/aad6b4c4-517f-41a2-b986-e7367bef493b-kube-api-access-cg4cv\") pod \"maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.370807    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldb9k\" (UniqueName: \"kubernetes.io/projected/8b270123-f6a5-4202-b8e3-699b84d55ee9-kube-api-access-ldb9k\") pod \"maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.370856    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdqg\" (UniqueName: \"kubernetes.io/projected/c6d2cd1c-ddd7-419a-8507-7e355abaf111-kube-api-access-vrdqg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.370891    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zd9\" (UniqueName: \"kubernetes.io/projected/ee09f6bd-6a43-4113-9d5c-84029ec53140-kube-api-access-h5zd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.370925    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbg2\" (UniqueName: \"kubernetes.io/projected/b1265940-ee90-4ede-9e4a-9bd690eb1681-kube-api-access-5qbg2\") pod \"maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.370966    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4bx\" (UniqueName: \"kubernetes.io/projected/0489e1db-1b58-4955-966b-82a17015a4be-kube-api-access-nb4bx\") pod \"maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.371040    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee09f6bd-6a43-4113-9d5c-84029ec53140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.371119    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0489e1db-1b58-4955-966b-82a17015a4be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.371221    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1265940-ee90-4ede-9e4a-9bd690eb1681-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.371275    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d2cd1c-ddd7-419a-8507-7e355abaf111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.371314    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad6b4c4-517f-41a2-b986-e7367bef493b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.372195    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad6b4c4-517f-41a2-b986-e7367bef493b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.373617    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee09f6bd-6a43-4113-9d5c-84029ec53140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.380491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.374245    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0489e1db-1b58-4955-966b-82a17015a4be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.432701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.432647    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:15:00.432972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.432950    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"]
Oct 30 01:15:00.439283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.439254    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.440823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.440598    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"]
Oct 30 01:15:00.445110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.443911    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4cv\" (UniqueName: \"kubernetes.io/projected/aad6b4c4-517f-41a2-b986-e7367bef493b-kube-api-access-cg4cv\") pod \"maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.445110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.444849    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.446073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.446052    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4bx\" (UniqueName: \"kubernetes.io/projected/0489e1db-1b58-4955-966b-82a17015a4be-kube-api-access-nb4bx\") pod \"maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.447294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.447248    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zd9\" (UniqueName: \"kubernetes.io/projected/ee09f6bd-6a43-4113-9d5c-84029ec53140-kube-api-access-h5zd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.447748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.447512    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"]
Oct 30 01:15:00.447748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.447551    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"]
Oct 30 01:15:00.447748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.447631    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.453902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.453385    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"]
Oct 30 01:15:00.460656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.460530    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"]
Oct 30 01:15:00.515479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.514600    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"]
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516135    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj7nn\" (UniqueName: \"kubernetes.io/projected/8a5910f9-37eb-42cd-a09d-417340f02866-kube-api-access-kj7nn\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516193    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/ffea74c7-864a-4a9f-b52a-2edd8aea5559-kube-api-access-cpvjp\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516262    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78645a0a-feb8-41d3-95a2-addfb511c1fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516314    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1265940-ee90-4ede-9e4a-9bd690eb1681-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516342    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdz7\" (UniqueName: \"kubernetes.io/projected/78645a0a-feb8-41d3-95a2-addfb511c1fd-kube-api-access-bfdz7\") pod \"maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516394    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e08139-14a0-4499-aa79-6bb318c2f531-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516431    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d2cd1c-ddd7-419a-8507-7e355abaf111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516516    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5910f9-37eb-42cd-a09d-417340f02866-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516637    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffea74c7-864a-4a9f-b52a-2edd8aea5559-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516672    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b270123-f6a5-4202-b8e3-699b84d55ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516752    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26rs\" (UniqueName: \"kubernetes.io/projected/25e08139-14a0-4499-aa79-6bb318c2f531-kube-api-access-h26rs\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516823    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb9k\" (UniqueName: \"kubernetes.io/projected/8b270123-f6a5-4202-b8e3-699b84d55ee9-kube-api-access-ldb9k\") pod \"maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516876    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdqg\" (UniqueName: \"kubernetes.io/projected/c6d2cd1c-ddd7-419a-8507-7e355abaf111-kube-api-access-vrdqg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.516922    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbg2\" (UniqueName: \"kubernetes.io/projected/b1265940-ee90-4ede-9e4a-9bd690eb1681-kube-api-access-5qbg2\") pod \"maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.518531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.518355    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1265940-ee90-4ede-9e4a-9bd690eb1681-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.519502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.519246    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d2cd1c-ddd7-419a-8507-7e355abaf111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.520137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.520109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b270123-f6a5-4202-b8e3-699b84d55ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.522229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.521731    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"]
Oct 30 01:15:00.539203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.536077    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:00.539203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.535978    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"]
Oct 30 01:15:00.546102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.546075    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"]
Oct 30 01:15:00.547145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.547124    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:00.550617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.550589    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldb9k\" (UniqueName: \"kubernetes.io/projected/8b270123-f6a5-4202-b8e3-699b84d55ee9-kube-api-access-ldb9k\") pod \"maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.553251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.553223    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbg2\" (UniqueName: \"kubernetes.io/projected/b1265940-ee90-4ede-9e4a-9bd690eb1681-kube-api-access-5qbg2\") pod \"maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.558025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.558008    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"]
Oct 30 01:15:00.558140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.558131    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"]
Oct 30 01:15:00.558232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.558209    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:00.559360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.559334    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"]
Oct 30 01:15:00.560169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.560143    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdqg\" (UniqueName: \"kubernetes.io/projected/c6d2cd1c-ddd7-419a-8507-7e355abaf111-kube-api-access-vrdqg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.565908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.565850    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"]
Oct 30 01:15:00.566091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.566071    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:00.570345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.570309    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:00.600814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.600775    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:00.620722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.620515    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj7nn\" (UniqueName: \"kubernetes.io/projected/8a5910f9-37eb-42cd-a09d-417340f02866-kube-api-access-kj7nn\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.620943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.620918    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/ffea74c7-864a-4a9f-b52a-2edd8aea5559-kube-api-access-cpvjp\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.621340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.621319    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78645a0a-feb8-41d3-95a2-addfb511c1fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.621484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.621471    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdz7\" (UniqueName: \"kubernetes.io/projected/78645a0a-feb8-41d3-95a2-addfb511c1fd-kube-api-access-bfdz7\") pod \"maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.621597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.621582    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e08139-14a0-4499-aa79-6bb318c2f531-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.621732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.621721    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5910f9-37eb-42cd-a09d-417340f02866-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.621841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.621830    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v877h\" (UniqueName: \"kubernetes.io/projected/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-kube-api-access-v877h\") pod \"maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:00.621936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.621926    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293a0b39-9772-4823-b91c-8f99316e92db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:00.622036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.622026    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffea74c7-864a-4a9f-b52a-2edd8aea5559-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.622119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.622108    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59pn\" (UniqueName: \"kubernetes.io/projected/c0f66867-6e46-479b-a883-33b036fa6279-kube-api-access-j59pn\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:00.622238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.622223    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26rs\" (UniqueName: \"kubernetes.io/projected/25e08139-14a0-4499-aa79-6bb318c2f531-kube-api-access-h26rs\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.622399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.622382    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f66867-6e46-479b-a883-33b036fa6279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:00.622575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.622558    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67429\" (UniqueName: \"kubernetes.io/projected/a510c363-24f2-41f1-869e-a509b345d5e5-kube-api-access-67429\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:00.622731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.622714    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksppq\" (UniqueName: \"kubernetes.io/projected/293a0b39-9772-4823-b91c-8f99316e92db-kube-api-access-ksppq\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:00.622888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.622871    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:00.623119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.623101    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510c363-24f2-41f1-869e-a509b345d5e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:00.623280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.623261    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5910f9-37eb-42cd-a09d-417340f02866-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.624617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.624596    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78645a0a-feb8-41d3-95a2-addfb511c1fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.627560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.624723    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e08139-14a0-4499-aa79-6bb318c2f531-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.627673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.627651    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"]
Oct 30 01:15:00.628448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.625464    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffea74c7-864a-4a9f-b52a-2edd8aea5559-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.628593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.628478    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:00.631347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.631320    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:00.636088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.636073    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"]
Oct 30 01:15:00.638519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.636244    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:00.649516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.649124    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:00.658535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.658395    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdz7\" (UniqueName: \"kubernetes.io/projected/78645a0a-feb8-41d3-95a2-addfb511c1fd-kube-api-access-bfdz7\") pod \"maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.659158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.659129    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj7nn\" (UniqueName: \"kubernetes.io/projected/8a5910f9-37eb-42cd-a09d-417340f02866-kube-api-access-kj7nn\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.659286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.659266    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/ffea74c7-864a-4a9f-b52a-2edd8aea5559-kube-api-access-cpvjp\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.659874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.659841    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26rs\" (UniqueName: \"kubernetes.io/projected/25e08139-14a0-4499-aa79-6bb318c2f531-kube-api-access-h26rs\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.668715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.668691    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"]
Oct 30 01:15:00.668926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.668913    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"]
Oct 30 01:15:00.714530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.703665    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:00.733868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.733829    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:00.733985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.733897    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599qq\" (UniqueName: \"kubernetes.io/projected/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-kube-api-access-599qq\") pod \"maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:00.734088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.734071    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkfrk\" (UniqueName: \"kubernetes.io/projected/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-kube-api-access-tkfrk\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:00.734287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.734268    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v877h\" (UniqueName: \"kubernetes.io/projected/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-kube-api-access-v877h\") pod \"maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:00.734360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.734341    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293a0b39-9772-4823-b91c-8f99316e92db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:00.734571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.734471    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59pn\" (UniqueName: \"kubernetes.io/projected/c0f66867-6e46-479b-a883-33b036fa6279-kube-api-access-j59pn\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:00.735544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.735515    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:00.737187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.735846    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f66867-6e46-479b-a883-33b036fa6279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:00.737187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.735932    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67429\" (UniqueName: \"kubernetes.io/projected/a510c363-24f2-41f1-869e-a509b345d5e5-kube-api-access-67429\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:00.737187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.735994    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksppq\" (UniqueName: \"kubernetes.io/projected/293a0b39-9772-4823-b91c-8f99316e92db-kube-api-access-ksppq\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:00.737187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.736069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:00.737187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.736197    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510c363-24f2-41f1-869e-a509b345d5e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:00.743370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.737891    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:00.743370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.738724    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f66867-6e46-479b-a883-33b036fa6279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:00.743370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.739913    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510c363-24f2-41f1-869e-a509b345d5e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:00.746333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.743719    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:00.757290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.756594    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293a0b39-9772-4823-b91c-8f99316e92db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:00.838839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.837228    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59pn\" (UniqueName: \"kubernetes.io/projected/c0f66867-6e46-479b-a883-33b036fa6279-kube-api-access-j59pn\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:00.842841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.842806    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"]
Oct 30 01:15:00.852100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.852069    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:00.855924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.855893    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"]
Oct 30 01:15:00.856154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.856138    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"]
Oct 30 01:15:00.856426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.856408    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:00.860527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.858141    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v877h\" (UniqueName: \"kubernetes.io/projected/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-kube-api-access-v877h\") pod \"maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:00.864153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.864131    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.866638    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.866703    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.867030    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6xf4\" (UniqueName: \"kubernetes.io/projected/5b931cfc-6a3d-4381-b5af-035ea30563cb-kube-api-access-r6xf4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.867092    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.867143    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b931cfc-6a3d-4381-b5af-035ea30563cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.867412    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.867446    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599qq\" (UniqueName: \"kubernetes.io/projected/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-kube-api-access-599qq\") pod \"maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:00.869246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.867549    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkfrk\" (UniqueName: \"kubernetes.io/projected/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-kube-api-access-tkfrk\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:00.872369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.872300    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:00.894386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.891213    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:00.938575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.938357    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:00.954858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.954829    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"]
Oct 30 01:15:00.955362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.955029    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67429\" (UniqueName: \"kubernetes.io/projected/a510c363-24f2-41f1-869e-a509b345d5e5-kube-api-access-67429\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:00.955912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.955888    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkfrk\" (UniqueName: \"kubernetes.io/projected/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-kube-api-access-tkfrk\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:00.969944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.969868    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599qq\" (UniqueName: \"kubernetes.io/projected/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-kube-api-access-599qq\") pod \"maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:01.002068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.997288    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksppq\" (UniqueName: \"kubernetes.io/projected/293a0b39-9772-4823-b91c-8f99316e92db-kube-api-access-ksppq\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:01.002068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.998997    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6xf4\" (UniqueName: \"kubernetes.io/projected/5b931cfc-6a3d-4381-b5af-035ea30563cb-kube-api-access-r6xf4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:01.002068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.999069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b931cfc-6a3d-4381-b5af-035ea30563cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:01.002068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:00.999124    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zl72\" (UniqueName: \"kubernetes.io/projected/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-kube-api-access-4zl72\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:01.015158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.014516    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b931cfc-6a3d-4381-b5af-035ea30563cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:01.017246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.016607    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:01.027689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.027504    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:01.035300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.034775    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"]
Oct 30 01:15:01.042816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.042733    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"]
Oct 30 01:15:01.044999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.044788    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:01.057755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.057566    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:01.061621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.061265    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"]
Oct 30 01:15:01.077544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.072939    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:01.129553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.127988    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zl72\" (UniqueName: \"kubernetes.io/projected/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-kube-api-access-4zl72\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:01.129553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.128107    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:01.135467 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.131049    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:01.142623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.142538    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"]
Oct 30 01:15:01.145279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.145222    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6xf4\" (UniqueName: \"kubernetes.io/projected/5b931cfc-6a3d-4381-b5af-035ea30563cb-kube-api-access-r6xf4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:01.150293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.150265    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zl72\" (UniqueName: \"kubernetes.io/projected/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-kube-api-access-4zl72\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:01.165718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.165627    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"]
Oct 30 01:15:01.203168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.203131    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:01.236783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.225606    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"]
Oct 30 01:15:01.340498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.338594    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:01.348873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.348789    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:01.518605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.517948    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:01.856783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.843839    5104 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 30 01:15:01.856783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:01.843916    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 30 01:15:02.245318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.245289    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"]
Oct 30 01:15:02.316320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.297564    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"]
Oct 30 01:15:02.362253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.362218    5104 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 30 01:15:02.362385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.362282    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 30 01:15:02.414381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.414344    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"]
Oct 30 01:15:02.423751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.423713    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"]
Oct 30 01:15:02.429274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.429242    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" event={"ID":"0489e1db-1b58-4955-966b-82a17015a4be","Type":"ContainerStarted","Data":"d6c418c5e4f5ecc6322bb007c192d91f30ff71487a3fae9ee11fa87f4eb513cf"}
Oct 30 01:15:02.431775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.431739    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" event={"ID":"ee09f6bd-6a43-4113-9d5c-84029ec53140","Type":"ContainerStarted","Data":"d728f4cc898a1d6acfc0a97ecb3be07e0b0477f715697bf82a8255fed8406430"}
Oct 30 01:15:02.725955 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:02.706013    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffea74c7_864a_4a9f_b52a_2edd8aea5559.slice/crio-89af54181e3048c880230eeba94bca7afdafff6bb93246ab2df50e0111ce5bae WatchSource:0}: Error finding container 89af54181e3048c880230eeba94bca7afdafff6bb93246ab2df50e0111ce5bae: Status 404 returned error can't find the container with id 89af54181e3048c880230eeba94bca7afdafff6bb93246ab2df50e0111ce5bae
Oct 30 01:15:02.733633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.733593    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"]
Oct 30 01:15:02.734919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.734697    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"]
Oct 30 01:15:02.734919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.734738    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"]
Oct 30 01:15:02.744365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.744278    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"]
Oct 30 01:15:02.753595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.753567    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"]
Oct 30 01:15:02.762401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.762362    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"]
Oct 30 01:15:02.835188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:02.835152    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"]
Oct 30 01:15:02.848912 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:02.848883    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5910f9_37eb_42cd_a09d_417340f02866.slice/crio-292346fd3376ec12ffbb44092fc188551bdce6749d2e3e61cc916836544c14b7 WatchSource:0}: Error finding container 292346fd3376ec12ffbb44092fc188551bdce6749d2e3e61cc916836544c14b7: Status 404 returned error can't find the container with id 292346fd3376ec12ffbb44092fc188551bdce6749d2e3e61cc916836544c14b7
Oct 30 01:15:03.058760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.058706    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"]
Oct 30 01:15:03.270366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.270332    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"]
Oct 30 01:15:03.271768 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:03.271734    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6a9e4d_59c8_4aca_a777_c5f2510b1d16.slice/crio-40e253a918f313e6313a93fbc89bcef2e75a5acb0c946779f0fac5a912665884 WatchSource:0}: Error finding container 40e253a918f313e6313a93fbc89bcef2e75a5acb0c946779f0fac5a912665884: Status 404 returned error can't find the container with id 40e253a918f313e6313a93fbc89bcef2e75a5acb0c946779f0fac5a912665884
Oct 30 01:15:03.318555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.315757    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"]
Oct 30 01:15:03.322210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.322184    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"]
Oct 30 01:15:03.330601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.330559    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"]
Oct 30 01:15:03.333252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.333231    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"]
Oct 30 01:15:03.338805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.338773    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"]
Oct 30 01:15:03.361988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.361890    5104 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 30 01:15:03.362175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.361979    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 30 01:15:03.442537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.442439    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" event={"ID":"aad6b4c4-517f-41a2-b986-e7367bef493b","Type":"ContainerStarted","Data":"863f68ca263a810000f8a87796686127d66630977ce31ddfaa6fde888a169b97"}
Oct 30 01:15:03.443982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.443952    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerStarted","Data":"629ff58eb44cfc6c4fe3a13b4ccc90e8d8748367d9dc554eb82eea1cee0aba4c"}
Oct 30 01:15:03.445410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.445385    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" event={"ID":"4cbf3f29-d8b1-4215-8c7d-b7311a0df231","Type":"ContainerStarted","Data":"80dcd7078d44ba93d2524023b80d7a240e0a0434fbd1511927c83cf1e5318b86"}
Oct 30 01:15:03.446916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.446892    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" event={"ID":"8a5910f9-37eb-42cd-a09d-417340f02866","Type":"ContainerStarted","Data":"292346fd3376ec12ffbb44092fc188551bdce6749d2e3e61cc916836544c14b7"}
Oct 30 01:15:03.454539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.454508    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerStarted","Data":"9512caaa1a925ae92a4d6be0eaffd2b1fe88624c9be8c5fb449ded71d78419e2"}
Oct 30 01:15:03.457001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.456974    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" event={"ID":"0489e1db-1b58-4955-966b-82a17015a4be","Type":"ContainerStarted","Data":"98cb8048e694ef9a150e297da32411120d135bbae39808e2f40b4da800f27ce6"}
Oct 30 01:15:03.458965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.458941    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" event={"ID":"c0f66867-6e46-479b-a883-33b036fa6279","Type":"ContainerStarted","Data":"c121382d1c80367f27db3ac49d82fd8c5c784d785c67a46e9c320ab15a241707"}
Oct 30 01:15:03.460617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.460590    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" event={"ID":"8b270123-f6a5-4202-b8e3-699b84d55ee9","Type":"ContainerStarted","Data":"4fb7220e53af08a96645c7d63f6965b47b244da636716d15ceef4836c47f197c"}
Oct 30 01:15:03.462493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.462396    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" event={"ID":"ee09f6bd-6a43-4113-9d5c-84029ec53140","Type":"ContainerStarted","Data":"7d21c73d95d08db08dc45d16e9f04cbe0e3e2c4cf19967a91263331c30b24749"}
Oct 30 01:15:03.464419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.464398    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" event={"ID":"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16","Type":"ContainerStarted","Data":"40e253a918f313e6313a93fbc89bcef2e75a5acb0c946779f0fac5a912665884"}
Oct 30 01:15:03.465897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.465871    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" event={"ID":"25e08139-14a0-4499-aa79-6bb318c2f531","Type":"ContainerStarted","Data":"e7475e95da47d5fd6e25d6f2f3ba1520d00feecb62d71479213fd95fcf286db9"}
Oct 30 01:15:03.467182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.467160    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" event={"ID":"78645a0a-feb8-41d3-95a2-addfb511c1fd","Type":"ContainerStarted","Data":"2e758c0a7564f716c4b4bb50fb9528c9c1cdac9f3bb56285d4ca144f933aadcc"}
Oct 30 01:15:03.468382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.468360    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" event={"ID":"ffea74c7-864a-4a9f-b52a-2edd8aea5559","Type":"ContainerStarted","Data":"89af54181e3048c880230eeba94bca7afdafff6bb93246ab2df50e0111ce5bae"}
Oct 30 01:15:03.518727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.518659    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" podStartSLOduration=3.518632974 podStartE2EDuration="3.518632974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.49710247 +0000 UTC m=+1945.044792991" watchObservedRunningTime="2026-10-30 01:15:03.518632974 +0000 UTC m=+1945.066323476"
Oct 30 01:15:03.528723 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:03.528676    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b931cfc_6a3d_4381_b5af_035ea30563cb.slice/crio-2137778dc7d3b203f8702762a6bb9ac32c156e16ac126aac4a58d51f2ed44017 WatchSource:0}: Error finding container 2137778dc7d3b203f8702762a6bb9ac32c156e16ac126aac4a58d51f2ed44017: Status 404 returned error can't find the container with id 2137778dc7d3b203f8702762a6bb9ac32c156e16ac126aac4a58d51f2ed44017
Oct 30 01:15:03.546756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:03.546714    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" podStartSLOduration=3.546698229 podStartE2EDuration="3.546698229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.535068918 +0000 UTC m=+1945.082759411" watchObservedRunningTime="2026-10-30 01:15:03.546698229 +0000 UTC m=+1945.094388726"
Oct 30 01:15:04.361921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.361868    5104 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 30 01:15:04.362368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.361938    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 30 01:15:04.518081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.518047    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" event={"ID":"8a5910f9-37eb-42cd-a09d-417340f02866","Type":"ContainerStarted","Data":"cde7154067a30b37b08a9277de7d25ddbfcac9f821b248ff30f1d936e3373448"}
Oct 30 01:15:04.535450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.533478    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" event={"ID":"5b931cfc-6a3d-4381-b5af-035ea30563cb","Type":"ContainerStarted","Data":"2137778dc7d3b203f8702762a6bb9ac32c156e16ac126aac4a58d51f2ed44017"}
Oct 30 01:15:04.543809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.542513    5104 generic.go:356] "Generic (PLEG): container finished" podID="0489e1db-1b58-4955-966b-82a17015a4be" containerID="98cb8048e694ef9a150e297da32411120d135bbae39808e2f40b4da800f27ce6" exitCode=60
Oct 30 01:15:04.543809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.542602    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" event={"ID":"0489e1db-1b58-4955-966b-82a17015a4be","Type":"ContainerDied","Data":"98cb8048e694ef9a150e297da32411120d135bbae39808e2f40b4da800f27ce6"}
Oct 30 01:15:04.543809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.542952    5104 scope.go:122] "RemoveContainer" containerID="98cb8048e694ef9a150e297da32411120d135bbae39808e2f40b4da800f27ce6"
Oct 30 01:15:04.544773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.544750    5104 generic.go:356] "Generic (PLEG): container finished" podID="c0f66867-6e46-479b-a883-33b036fa6279" containerID="ac0d05504979c8881378715aac9e2406130c7c4df47aceafbf04ae63fb29f56d" exitCode=60
Oct 30 01:15:04.545082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.545068    5104 scope.go:122] "RemoveContainer" containerID="ac0d05504979c8881378715aac9e2406130c7c4df47aceafbf04ae63fb29f56d"
Oct 30 01:15:04.545201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.545176    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" event={"ID":"c0f66867-6e46-479b-a883-33b036fa6279","Type":"ContainerDied","Data":"ac0d05504979c8881378715aac9e2406130c7c4df47aceafbf04ae63fb29f56d"}
Oct 30 01:15:04.546929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.546904    5104 generic.go:356] "Generic (PLEG): container finished" podID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerID="eb572cc3e3e19896e8fff43399ea76b3969117486f5d6941a9c48d14a05975d6" exitCode=60
Oct 30 01:15:04.547022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.546981    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" event={"ID":"8b270123-f6a5-4202-b8e3-699b84d55ee9","Type":"ContainerDied","Data":"eb572cc3e3e19896e8fff43399ea76b3969117486f5d6941a9c48d14a05975d6"}
Oct 30 01:15:04.547254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.547236    5104 scope.go:122] "RemoveContainer" containerID="eb572cc3e3e19896e8fff43399ea76b3969117486f5d6941a9c48d14a05975d6"
Oct 30 01:15:04.548691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.548668    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" event={"ID":"a510c363-24f2-41f1-869e-a509b345d5e5","Type":"ContainerStarted","Data":"ce1b214d1fbef649a995bf00168e0fdcd2e3be4674cc359bf58c0fd80ad17ece"}
Oct 30 01:15:04.548797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.548710    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" event={"ID":"a510c363-24f2-41f1-869e-a509b345d5e5","Type":"ContainerStarted","Data":"32bd5182523d6f3f63a6cd643326c5357c71b29fea9300ef721026e2f9056a6d"}
Oct 30 01:15:04.550243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.550221    5104 generic.go:356] "Generic (PLEG): container finished" podID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerID="7d21c73d95d08db08dc45d16e9f04cbe0e3e2c4cf19967a91263331c30b24749" exitCode=60
Oct 30 01:15:04.550328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.550293    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" event={"ID":"ee09f6bd-6a43-4113-9d5c-84029ec53140","Type":"ContainerDied","Data":"7d21c73d95d08db08dc45d16e9f04cbe0e3e2c4cf19967a91263331c30b24749"}
Oct 30 01:15:04.550530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.550512    5104 scope.go:122] "RemoveContainer" containerID="7d21c73d95d08db08dc45d16e9f04cbe0e3e2c4cf19967a91263331c30b24749"
Oct 30 01:15:04.551645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.551626    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" event={"ID":"f0ae7c6c-84f1-47ab-8b4e-86a121949b97","Type":"ContainerStarted","Data":"2d3768466b2bf72d7f57792906d7bb597f6990aee6ddcf6168a590844b1b8994"}
Oct 30 01:15:04.553330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.553308    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" event={"ID":"25e08139-14a0-4499-aa79-6bb318c2f531","Type":"ContainerStarted","Data":"f5442cb81d1925398a295630736120b7e7dafe98c7f8e9c7d7cbda96bf93b254"}
Oct 30 01:15:04.554874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.554852    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerStarted","Data":"4733e114d5a3ed4fa50a0568812af48f1f5e277ecc7a01c838c3629acd3f51d9"}
Oct 30 01:15:04.556751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.556708    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" podStartSLOduration=4.55669288 podStartE2EDuration="4.55669288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.554194749 +0000 UTC m=+1946.101885246" watchObservedRunningTime="2026-10-30 01:15:04.55669288 +0000 UTC m=+1946.104383377"
Oct 30 01:15:04.557381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.557363    5104 generic.go:356] "Generic (PLEG): container finished" podID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerID="a7791296f4e3c86866efeee4cadbcae1610608c8cb5fac37b770e97de27636cc" exitCode=60
Oct 30 01:15:04.557622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.557594    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" event={"ID":"aad6b4c4-517f-41a2-b986-e7367bef493b","Type":"ContainerDied","Data":"a7791296f4e3c86866efeee4cadbcae1610608c8cb5fac37b770e97de27636cc"}
Oct 30 01:15:04.558199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.558170    5104 scope.go:122] "RemoveContainer" containerID="a7791296f4e3c86866efeee4cadbcae1610608c8cb5fac37b770e97de27636cc"
Oct 30 01:15:04.559449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.559428    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" event={"ID":"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93","Type":"ContainerStarted","Data":"b137837464679a5f03d017e8ad5c0c5258e88aef0b62e102f8db212ca6388560"}
Oct 30 01:15:04.661988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.661887    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" podStartSLOduration=4.661872125 podStartE2EDuration="4.661872125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.656031716 +0000 UTC m=+1946.203722212" watchObservedRunningTime="2026-10-30 01:15:04.661872125 +0000 UTC m=+1946.209562617"
Oct 30 01:15:04.723154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:04.723108    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" podStartSLOduration=4.723092545 podStartE2EDuration="4.723092545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.71629154 +0000 UTC m=+1946.263982035" watchObservedRunningTime="2026-10-30 01:15:04.723092545 +0000 UTC m=+1946.270783041"
Oct 30 01:15:05.115279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.112957    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:15:05.362550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.362192    5104 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 30 01:15:05.362878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.362580    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 30 01:15:05.633426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.633391    5104 generic.go:356] "Generic (PLEG): container finished" podID="a510c363-24f2-41f1-869e-a509b345d5e5" containerID="ce1b214d1fbef649a995bf00168e0fdcd2e3be4674cc359bf58c0fd80ad17ece" exitCode=60
Oct 30 01:15:05.633669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.633645    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" event={"ID":"a510c363-24f2-41f1-869e-a509b345d5e5","Type":"ContainerDied","Data":"ce1b214d1fbef649a995bf00168e0fdcd2e3be4674cc359bf58c0fd80ad17ece"}
Oct 30 01:15:05.634225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.634198    5104 scope.go:122] "RemoveContainer" containerID="ce1b214d1fbef649a995bf00168e0fdcd2e3be4674cc359bf58c0fd80ad17ece"
Oct 30 01:15:05.637903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.637526    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" event={"ID":"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16","Type":"ContainerStarted","Data":"ef74e323ab623199ae0d0822a094b9199fe139e52c8c0207b248103747c413f6"}
Oct 30 01:15:05.637903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.637761    5104 scope.go:122] "RemoveContainer" containerID="ef74e323ab623199ae0d0822a094b9199fe139e52c8c0207b248103747c413f6"
Oct 30 01:15:05.650737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.647818    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" event={"ID":"f0ae7c6c-84f1-47ab-8b4e-86a121949b97","Type":"ContainerStarted","Data":"9598076129e6d5ae27d6cd579b1990c4c2aae25572be94b12582ef7b51e8984a"}
Oct 30 01:15:05.655341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.654264    5104 generic.go:356] "Generic (PLEG): container finished" podID="25e08139-14a0-4499-aa79-6bb318c2f531" containerID="f5442cb81d1925398a295630736120b7e7dafe98c7f8e9c7d7cbda96bf93b254" exitCode=60
Oct 30 01:15:05.655341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.654373    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" event={"ID":"25e08139-14a0-4499-aa79-6bb318c2f531","Type":"ContainerDied","Data":"f5442cb81d1925398a295630736120b7e7dafe98c7f8e9c7d7cbda96bf93b254"}
Oct 30 01:15:05.655341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.654886    5104 scope.go:122] "RemoveContainer" containerID="f5442cb81d1925398a295630736120b7e7dafe98c7f8e9c7d7cbda96bf93b254"
Oct 30 01:15:05.662322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.661684    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" event={"ID":"78645a0a-feb8-41d3-95a2-addfb511c1fd","Type":"ContainerStarted","Data":"ef0b8a25ab5dc812a21f711a3627ae12469cc991e6eb9865da7a909d484fa5e8"}
Oct 30 01:15:05.662322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.662008    5104 scope.go:122] "RemoveContainer" containerID="ef0b8a25ab5dc812a21f711a3627ae12469cc991e6eb9865da7a909d484fa5e8"
Oct 30 01:15:05.664650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.664621    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" event={"ID":"ffea74c7-864a-4a9f-b52a-2edd8aea5559","Type":"ContainerStarted","Data":"407a87241c6407c04471227c43fc0f4a893fef54f785ac84b768ed416fdde0fc"}
Oct 30 01:15:05.667858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.667837    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerStarted","Data":"c265c26fffd2d97c80224f93ad1d2072805d8c0b9430270d398e88623238377b"}
Oct 30 01:15:05.668220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.668199    5104 scope.go:122] "RemoveContainer" containerID="c265c26fffd2d97c80224f93ad1d2072805d8c0b9430270d398e88623238377b"
Oct 30 01:15:05.671775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.671746    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" event={"ID":"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93","Type":"ContainerStarted","Data":"46f2aec7abb6822e8a4ed558a8ab24714a90d249a646c50995bc653ca67063cc"}
Oct 30 01:15:05.672134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.672104    5104 scope.go:122] "RemoveContainer" containerID="46f2aec7abb6822e8a4ed558a8ab24714a90d249a646c50995bc653ca67063cc"
Oct 30 01:15:05.720134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.720093    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerStarted","Data":"1c5d6dbd7e9237122ed385f07943c045178070f2a1793ecb8676d28393ebce74"}
Oct 30 01:15:05.724214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.724178    5104 generic.go:356] "Generic (PLEG): container finished" podID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerID="d12351b50178e67bbf1f908f44cefa3421672d8ba0fc60fc121f1d2a3e6308ad" exitCode=60
Oct 30 01:15:05.724340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.724266    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" event={"ID":"4cbf3f29-d8b1-4215-8c7d-b7311a0df231","Type":"ContainerDied","Data":"d12351b50178e67bbf1f908f44cefa3421672d8ba0fc60fc121f1d2a3e6308ad"}
Oct 30 01:15:05.725197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.724919    5104 scope.go:122] "RemoveContainer" containerID="d12351b50178e67bbf1f908f44cefa3421672d8ba0fc60fc121f1d2a3e6308ad"
Oct 30 01:15:05.741507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.741140    5104 generic.go:356] "Generic (PLEG): container finished" podID="8a5910f9-37eb-42cd-a09d-417340f02866" containerID="cde7154067a30b37b08a9277de7d25ddbfcac9f821b248ff30f1d936e3373448" exitCode=60
Oct 30 01:15:05.741507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.741238    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" event={"ID":"8a5910f9-37eb-42cd-a09d-417340f02866","Type":"ContainerDied","Data":"cde7154067a30b37b08a9277de7d25ddbfcac9f821b248ff30f1d936e3373448"}
Oct 30 01:15:05.741880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.741838    5104 scope.go:122] "RemoveContainer" containerID="cde7154067a30b37b08a9277de7d25ddbfcac9f821b248ff30f1d936e3373448"
Oct 30 01:15:05.748631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.748605    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" event={"ID":"5b931cfc-6a3d-4381-b5af-035ea30563cb","Type":"ContainerStarted","Data":"059e7b4c0ac1cc321ee3f925c722d9a014d7cb19a03252251234e9ea2726c72c"}
Oct 30 01:15:05.749358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.749146    5104 scope.go:122] "RemoveContainer" containerID="059e7b4c0ac1cc321ee3f925c722d9a014d7cb19a03252251234e9ea2726c72c"
Oct 30 01:15:05.751763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.750943    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerStarted","Data":"8491a6c342584f21ac94f95d2c9b65eba4b010115371db013e69b4544563c12a"}
Oct 30 01:15:05.751763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.751310    5104 scope.go:122] "RemoveContainer" containerID="8491a6c342584f21ac94f95d2c9b65eba4b010115371db013e69b4544563c12a"
Oct 30 01:15:05.868929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:05.867964    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" podStartSLOduration=5.867945562 podStartE2EDuration="5.867945562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:05.859758899 +0000 UTC m=+1947.407449394" watchObservedRunningTime="2026-10-30 01:15:05.867945562 +0000 UTC m=+1947.415636060"
Oct 30 01:15:06.015728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.015665    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" podStartSLOduration=6.015650799 podStartE2EDuration="6.015650799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.015196128 +0000 UTC m=+1947.562886623" watchObservedRunningTime="2026-10-30 01:15:06.015650799 +0000 UTC m=+1947.563341335"
Oct 30 01:15:06.198905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.198845    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" podStartSLOduration=6.198829132 podStartE2EDuration="6.198829132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.193505115 +0000 UTC m=+1947.741195613" watchObservedRunningTime="2026-10-30 01:15:06.198829132 +0000 UTC m=+1947.746519628"
Oct 30 01:15:06.367644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.367610    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:15:06.368803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.368781    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:15:06.767504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.767158    5104 generic.go:356] "Generic (PLEG): container finished" podID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerID="8491a6c342584f21ac94f95d2c9b65eba4b010115371db013e69b4544563c12a" exitCode=60
Oct 30 01:15:06.767504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.767329    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerDied","Data":"8491a6c342584f21ac94f95d2c9b65eba4b010115371db013e69b4544563c12a"}
Oct 30 01:15:06.767504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.767376    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerStarted","Data":"0f928e4a295a1096e3458debefe70f418db62944cdb852eb0bbd17ffd3aae748"}
Oct 30 01:15:06.770314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.770289    5104 generic.go:356] "Generic (PLEG): container finished" podID="0489e1db-1b58-4955-966b-82a17015a4be" containerID="551cfe6d635fe456d2b08334f60d95243bcc519f246783f5145726da12ab32d5" exitCode=60
Oct 30 01:15:06.770448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.770362    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" event={"ID":"0489e1db-1b58-4955-966b-82a17015a4be","Type":"ContainerDied","Data":"551cfe6d635fe456d2b08334f60d95243bcc519f246783f5145726da12ab32d5"}
Oct 30 01:15:06.770448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.770399    5104 scope.go:122] "RemoveContainer" containerID="98cb8048e694ef9a150e297da32411120d135bbae39808e2f40b4da800f27ce6"
Oct 30 01:15:06.770897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.770882    5104 scope.go:122] "RemoveContainer" containerID="551cfe6d635fe456d2b08334f60d95243bcc519f246783f5145726da12ab32d5"
Oct 30 01:15:06.771355 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:06.771330    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29_odh-ai-gateway-infra(0489e1db-1b58-4955-966b-82a17015a4be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" podUID="0489e1db-1b58-4955-966b-82a17015a4be"
Oct 30 01:15:06.784067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.784032    5104 generic.go:356] "Generic (PLEG): container finished" podID="c0f66867-6e46-479b-a883-33b036fa6279" containerID="251a46953ccfcb6942c5364d70fa9f636b166a1a014b92575e03f4df442b4249" exitCode=60
Oct 30 01:15:06.784226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.784127    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" event={"ID":"c0f66867-6e46-479b-a883-33b036fa6279","Type":"ContainerDied","Data":"251a46953ccfcb6942c5364d70fa9f636b166a1a014b92575e03f4df442b4249"}
Oct 30 01:15:06.784625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.784604    5104 scope.go:122] "RemoveContainer" containerID="251a46953ccfcb6942c5364d70fa9f636b166a1a014b92575e03f4df442b4249"
Oct 30 01:15:06.785000 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:06.784975    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97_odh-ai-gateway-infra(c0f66867-6e46-479b-a883-33b036fa6279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" podUID="c0f66867-6e46-479b-a883-33b036fa6279"
Oct 30 01:15:06.790483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.790099    5104 generic.go:356] "Generic (PLEG): container finished" podID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerID="8d9b550b636c4019951ee9f13b9857f2c041193e4845edfc646891ce2d71f486" exitCode=60
Oct 30 01:15:06.790483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.790277    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" event={"ID":"8b270123-f6a5-4202-b8e3-699b84d55ee9","Type":"ContainerDied","Data":"8d9b550b636c4019951ee9f13b9857f2c041193e4845edfc646891ce2d71f486"}
Oct 30 01:15:06.791686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.790885    5104 scope.go:122] "RemoveContainer" containerID="8d9b550b636c4019951ee9f13b9857f2c041193e4845edfc646891ce2d71f486"
Oct 30 01:15:06.791686 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:06.791292    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7_odh-ai-gateway-infra(8b270123-f6a5-4202-b8e3-699b84d55ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9"
Oct 30 01:15:06.794447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.794373    5104 generic.go:356] "Generic (PLEG): container finished" podID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerID="9fc49d66a8d02a0b3bc9bc5a1f7b43eff6faae606d2b45d53997f761c32b97d9" exitCode=60
Oct 30 01:15:06.794614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.794482    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" event={"ID":"ee09f6bd-6a43-4113-9d5c-84029ec53140","Type":"ContainerDied","Data":"9fc49d66a8d02a0b3bc9bc5a1f7b43eff6faae606d2b45d53997f761c32b97d9"}
Oct 30 01:15:06.795340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.794916    5104 scope.go:122] "RemoveContainer" containerID="9fc49d66a8d02a0b3bc9bc5a1f7b43eff6faae606d2b45d53997f761c32b97d9"
Oct 30 01:15:06.795340 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:06.795283    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx_odh-ai-gateway-infra(ee09f6bd-6a43-4113-9d5c-84029ec53140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140"
Oct 30 01:15:06.798316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.798291    5104 generic.go:356] "Generic (PLEG): container finished" podID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerID="ef74e323ab623199ae0d0822a094b9199fe139e52c8c0207b248103747c413f6" exitCode=60
Oct 30 01:15:06.798427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.798336    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" event={"ID":"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16","Type":"ContainerDied","Data":"ef74e323ab623199ae0d0822a094b9199fe139e52c8c0207b248103747c413f6"}
Oct 30 01:15:06.801346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.801320    5104 generic.go:356] "Generic (PLEG): container finished" podID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerID="9598076129e6d5ae27d6cd579b1990c4c2aae25572be94b12582ef7b51e8984a" exitCode=60
Oct 30 01:15:06.801471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.801377    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" event={"ID":"f0ae7c6c-84f1-47ab-8b4e-86a121949b97","Type":"ContainerDied","Data":"9598076129e6d5ae27d6cd579b1990c4c2aae25572be94b12582ef7b51e8984a"}
Oct 30 01:15:06.801775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.801755    5104 scope.go:122] "RemoveContainer" containerID="9598076129e6d5ae27d6cd579b1990c4c2aae25572be94b12582ef7b51e8984a"
Oct 30 01:15:06.801953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.801935    5104 scope.go:122] "RemoveContainer" containerID="ac0d05504979c8881378715aac9e2406130c7c4df47aceafbf04ae63fb29f56d"
Oct 30 01:15:06.804642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.804616    5104 generic.go:356] "Generic (PLEG): container finished" podID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerID="ef0b8a25ab5dc812a21f711a3627ae12469cc991e6eb9865da7a909d484fa5e8" exitCode=60
Oct 30 01:15:06.804743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.804704    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" event={"ID":"78645a0a-feb8-41d3-95a2-addfb511c1fd","Type":"ContainerDied","Data":"ef0b8a25ab5dc812a21f711a3627ae12469cc991e6eb9865da7a909d484fa5e8"}
Oct 30 01:15:06.807885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.807842    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" podStartSLOduration=6.807828677 podStartE2EDuration="6.807828677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.804273363 +0000 UTC m=+1948.351963859" watchObservedRunningTime="2026-10-30 01:15:06.807828677 +0000 UTC m=+1948.355519172"
Oct 30 01:15:06.810751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.809414    5104 generic.go:356] "Generic (PLEG): container finished" podID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerID="407a87241c6407c04471227c43fc0f4a893fef54f785ac84b768ed416fdde0fc" exitCode=60
Oct 30 01:15:06.810751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.809544    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" event={"ID":"ffea74c7-864a-4a9f-b52a-2edd8aea5559","Type":"ContainerDied","Data":"407a87241c6407c04471227c43fc0f4a893fef54f785ac84b768ed416fdde0fc"}
Oct 30 01:15:06.810751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.810004    5104 scope.go:122] "RemoveContainer" containerID="407a87241c6407c04471227c43fc0f4a893fef54f785ac84b768ed416fdde0fc"
Oct 30 01:15:06.813578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.813026    5104 generic.go:356] "Generic (PLEG): container finished" podID="293a0b39-9772-4823-b91c-8f99316e92db" containerID="c265c26fffd2d97c80224f93ad1d2072805d8c0b9430270d398e88623238377b" exitCode=60
Oct 30 01:15:06.813578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.813117    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerDied","Data":"c265c26fffd2d97c80224f93ad1d2072805d8c0b9430270d398e88623238377b"}
Oct 30 01:15:06.819260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.817441    5104 generic.go:356] "Generic (PLEG): container finished" podID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerID="568872b502a78c844b9ebfc81c40c44c4620de04a2b34d97e936956171a36e03" exitCode=60
Oct 30 01:15:06.819260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.817561    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" event={"ID":"aad6b4c4-517f-41a2-b986-e7367bef493b","Type":"ContainerDied","Data":"568872b502a78c844b9ebfc81c40c44c4620de04a2b34d97e936956171a36e03"}
Oct 30 01:15:06.819260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.818052    5104 scope.go:122] "RemoveContainer" containerID="568872b502a78c844b9ebfc81c40c44c4620de04a2b34d97e936956171a36e03"
Oct 30 01:15:06.819260 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:06.818502    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l_odh-ai-gateway-infra(aad6b4c4-517f-41a2-b986-e7367bef493b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b"
Oct 30 01:15:06.821563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.821536    5104 generic.go:356] "Generic (PLEG): container finished" podID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerID="46f2aec7abb6822e8a4ed558a8ab24714a90d249a646c50995bc653ca67063cc" exitCode=60
Oct 30 01:15:06.821721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.821688    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" event={"ID":"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93","Type":"ContainerDied","Data":"46f2aec7abb6822e8a4ed558a8ab24714a90d249a646c50995bc653ca67063cc"}
Oct 30 01:15:06.821803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.821731    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" event={"ID":"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93","Type":"ContainerStarted","Data":"454cf7a8ecdb9542c11d92e460c355b29f9e76f91b10fa0c9ff6b70537fdf67d"}
Oct 30 01:15:06.824318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.824292    5104 generic.go:356] "Generic (PLEG): container finished" podID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerID="1c5d6dbd7e9237122ed385f07943c045178070f2a1793ecb8676d28393ebce74" exitCode=60
Oct 30 01:15:06.824424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.824407    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerDied","Data":"1c5d6dbd7e9237122ed385f07943c045178070f2a1793ecb8676d28393ebce74"}
Oct 30 01:15:06.825168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.825147    5104 scope.go:122] "RemoveContainer" containerID="1c5d6dbd7e9237122ed385f07943c045178070f2a1793ecb8676d28393ebce74"
Oct 30 01:15:06.826633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.826615    5104 generic.go:356] "Generic (PLEG): container finished" podID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerID="059e7b4c0ac1cc321ee3f925c722d9a014d7cb19a03252251234e9ea2726c72c" exitCode=60
Oct 30 01:15:06.827260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.827242    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" event={"ID":"5b931cfc-6a3d-4381-b5af-035ea30563cb","Type":"ContainerDied","Data":"059e7b4c0ac1cc321ee3f925c722d9a014d7cb19a03252251234e9ea2726c72c"}
Oct 30 01:15:06.840115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.836172    5104 scope.go:122] "RemoveContainer" containerID="eb572cc3e3e19896e8fff43399ea76b3969117486f5d6941a9c48d14a05975d6"
Oct 30 01:15:06.865830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.865797    5104 scope.go:122] "RemoveContainer" containerID="7d21c73d95d08db08dc45d16e9f04cbe0e3e2c4cf19967a91263331c30b24749"
Oct 30 01:15:06.900990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.900968    5104 scope.go:122] "RemoveContainer" containerID="a7791296f4e3c86866efeee4cadbcae1610608c8cb5fac37b770e97de27636cc"
Oct 30 01:15:06.930588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:06.930526    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" podStartSLOduration=6.930502148 podStartE2EDuration="6.930502148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.925933269 +0000 UTC m=+1948.473623764" watchObservedRunningTime="2026-10-30 01:15:06.930502148 +0000 UTC m=+1948.478192641"
Oct 30 01:15:07.151781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.151723    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" podStartSLOduration=7.151707759 podStartE2EDuration="7.151707759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:07.15064884 +0000 UTC m=+1948.698339336" watchObservedRunningTime="2026-10-30 01:15:07.151707759 +0000 UTC m=+1948.699398249"
Oct 30 01:15:07.838254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.838221    5104 generic.go:356] "Generic (PLEG): container finished" podID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerID="0f928e4a295a1096e3458debefe70f418db62944cdb852eb0bbd17ffd3aae748" exitCode=60
Oct 30 01:15:07.838254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.838249    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerDied","Data":"0f928e4a295a1096e3458debefe70f418db62944cdb852eb0bbd17ffd3aae748"}
Oct 30 01:15:07.838847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.838341    5104 scope.go:122] "RemoveContainer" containerID="8491a6c342584f21ac94f95d2c9b65eba4b010115371db013e69b4544563c12a"
Oct 30 01:15:07.838847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.838786    5104 scope.go:122] "RemoveContainer" containerID="0f928e4a295a1096e3458debefe70f418db62944cdb852eb0bbd17ffd3aae748"
Oct 30 01:15:07.839264 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.839236    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg_odh-ai-gateway-infra(b1265940-ee90-4ede-9e4a-9bd690eb1681)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681"
Oct 30 01:15:07.840884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.840865    5104 scope.go:122] "RemoveContainer" containerID="551cfe6d635fe456d2b08334f60d95243bcc519f246783f5145726da12ab32d5"
Oct 30 01:15:07.841299 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.841274    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29_odh-ai-gateway-infra(0489e1db-1b58-4955-966b-82a17015a4be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" podUID="0489e1db-1b58-4955-966b-82a17015a4be"
Oct 30 01:15:07.842629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.842609    5104 scope.go:122] "RemoveContainer" containerID="251a46953ccfcb6942c5364d70fa9f636b166a1a014b92575e03f4df442b4249"
Oct 30 01:15:07.842990 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.842970    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97_odh-ai-gateway-infra(c0f66867-6e46-479b-a883-33b036fa6279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" podUID="c0f66867-6e46-479b-a883-33b036fa6279"
Oct 30 01:15:07.844571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.844553    5104 scope.go:122] "RemoveContainer" containerID="8d9b550b636c4019951ee9f13b9857f2c041193e4845edfc646891ce2d71f486"
Oct 30 01:15:07.844884 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.844859    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7_odh-ai-gateway-infra(8b270123-f6a5-4202-b8e3-699b84d55ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9"
Oct 30 01:15:07.846445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.846424    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" event={"ID":"a510c363-24f2-41f1-869e-a509b345d5e5","Type":"ContainerStarted","Data":"66e4e9df2f7f21b9b9a6a0aed7e05012da63a146b8d56abf1cf8304e4c665a39"}
Oct 30 01:15:07.848330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.848314    5104 scope.go:122] "RemoveContainer" containerID="9fc49d66a8d02a0b3bc9bc5a1f7b43eff6faae606d2b45d53997f761c32b97d9"
Oct 30 01:15:07.848637 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.848608    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx_odh-ai-gateway-infra(ee09f6bd-6a43-4113-9d5c-84029ec53140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140"
Oct 30 01:15:07.850123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.850099    5104 generic.go:356] "Generic (PLEG): container finished" podID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerID="9d2ab12ba29f814381c9dfb47dae47f15104f78d0f5e55f2e926f069a9ad3b6b" exitCode=60
Oct 30 01:15:07.850232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.850174    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" event={"ID":"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16","Type":"ContainerDied","Data":"9d2ab12ba29f814381c9dfb47dae47f15104f78d0f5e55f2e926f069a9ad3b6b"}
Oct 30 01:15:07.850396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.850381    5104 scope.go:122] "RemoveContainer" containerID="9d2ab12ba29f814381c9dfb47dae47f15104f78d0f5e55f2e926f069a9ad3b6b"
Oct 30 01:15:07.850689 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.850663    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c_odh-ai-gateway-infra(1e6a9e4d-59c8-4aca-a777-c5f2510b1d16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16"
Oct 30 01:15:07.852380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.852344    5104 generic.go:356] "Generic (PLEG): container finished" podID="25e08139-14a0-4499-aa79-6bb318c2f531" containerID="32f8d6cfba3993182622ccbb81ffa910bcdc344f15ca842bab26294686ae0c59" exitCode=60
Oct 30 01:15:07.852491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.852419    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" event={"ID":"25e08139-14a0-4499-aa79-6bb318c2f531","Type":"ContainerDied","Data":"32f8d6cfba3993182622ccbb81ffa910bcdc344f15ca842bab26294686ae0c59"}
Oct 30 01:15:07.852780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.852759    5104 scope.go:122] "RemoveContainer" containerID="32f8d6cfba3993182622ccbb81ffa910bcdc344f15ca842bab26294686ae0c59"
Oct 30 01:15:07.853090 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.853065    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz_odh-ai-gateway-infra(25e08139-14a0-4499-aa79-6bb318c2f531)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" podUID="25e08139-14a0-4499-aa79-6bb318c2f531"
Oct 30 01:15:07.855595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.855570    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" event={"ID":"78645a0a-feb8-41d3-95a2-addfb511c1fd","Type":"ContainerStarted","Data":"0c56ce3373e6c805a1033701ccf8fc7bfabb6227c23de6a590925555c854e2af"}
Oct 30 01:15:07.857639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.857621    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" event={"ID":"ffea74c7-864a-4a9f-b52a-2edd8aea5559","Type":"ContainerStarted","Data":"c8c0151c84a35a2a84895b0caf892a850793da9acc2f5f3f44ff93a216da0f50"}
Oct 30 01:15:07.859741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.859712    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerStarted","Data":"f28c40f9d7db2928b680b6f61acae80befe88c9259a9e5ff555a4406b4bebb9d"}
Oct 30 01:15:07.861617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.861598    5104 scope.go:122] "RemoveContainer" containerID="568872b502a78c844b9ebfc81c40c44c4620de04a2b34d97e936956171a36e03"
Oct 30 01:15:07.861900 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.861877    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l_odh-ai-gateway-infra(aad6b4c4-517f-41a2-b986-e7367bef493b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b"
Oct 30 01:15:07.863720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.863688    5104 generic.go:356] "Generic (PLEG): container finished" podID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerID="454cf7a8ecdb9542c11d92e460c355b29f9e76f91b10fa0c9ff6b70537fdf67d" exitCode=60
Oct 30 01:15:07.863830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.863767    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" event={"ID":"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93","Type":"ContainerDied","Data":"454cf7a8ecdb9542c11d92e460c355b29f9e76f91b10fa0c9ff6b70537fdf67d"}
Oct 30 01:15:07.864009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.863992    5104 scope.go:122] "RemoveContainer" containerID="454cf7a8ecdb9542c11d92e460c355b29f9e76f91b10fa0c9ff6b70537fdf67d"
Oct 30 01:15:07.864294 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.864273    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5_odh-ai-gateway-infra(45a664ef-0a32-4a1a-98a1-5ccfdb4eed93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93"
Oct 30 01:15:07.866089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.866069    5104 generic.go:356] "Generic (PLEG): container finished" podID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerID="20c3aab5ac3120a60a3daca5e8754ec4e9f02cb0a12fcb43c95a959e00c23796" exitCode=60
Oct 30 01:15:07.866172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.866131    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" event={"ID":"4cbf3f29-d8b1-4215-8c7d-b7311a0df231","Type":"ContainerDied","Data":"20c3aab5ac3120a60a3daca5e8754ec4e9f02cb0a12fcb43c95a959e00c23796"}
Oct 30 01:15:07.866427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.866407    5104 scope.go:122] "RemoveContainer" containerID="20c3aab5ac3120a60a3daca5e8754ec4e9f02cb0a12fcb43c95a959e00c23796"
Oct 30 01:15:07.866771 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.866743    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv_odh-ai-gateway-infra(4cbf3f29-d8b1-4215-8c7d-b7311a0df231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231"
Oct 30 01:15:07.868527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.868508    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" event={"ID":"8a5910f9-37eb-42cd-a09d-417340f02866","Type":"ContainerStarted","Data":"018f9e289c5f236da869f077ab3c69a71089be525b57a839ff98e704dd34b05c"}
Oct 30 01:15:07.870732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.870712    5104 generic.go:356] "Generic (PLEG): container finished" podID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerID="80638c9f9747572a90e8535612b84def5c89f4986fccafc550f3b29ba1c793c4" exitCode=60
Oct 30 01:15:07.870835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.870759    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" event={"ID":"5b931cfc-6a3d-4381-b5af-035ea30563cb","Type":"ContainerDied","Data":"80638c9f9747572a90e8535612b84def5c89f4986fccafc550f3b29ba1c793c4"}
Oct 30 01:15:07.871046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.871030    5104 scope.go:122] "RemoveContainer" containerID="80638c9f9747572a90e8535612b84def5c89f4986fccafc550f3b29ba1c793c4"
Oct 30 01:15:07.871313 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:07.871292    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l_odh-ai-gateway-infra(5b931cfc-6a3d-4381-b5af-035ea30563cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb"
Oct 30 01:15:07.910502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:07.910417    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" podStartSLOduration=7.910404669 podStartE2EDuration="7.910404669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:07.90995419 +0000 UTC m=+1949.457644687" watchObservedRunningTime="2026-10-30 01:15:07.910404669 +0000 UTC m=+1949.458095164"
Oct 30 01:15:08.368485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.357443    5104 scope.go:122] "RemoveContainer" containerID="ef74e323ab623199ae0d0822a094b9199fe139e52c8c0207b248103747c413f6"
Oct 30 01:15:08.490709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.490639    5104 scope.go:122] "RemoveContainer" containerID="f5442cb81d1925398a295630736120b7e7dafe98c7f8e9c7d7cbda96bf93b254"
Oct 30 01:15:08.511322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.511294    5104 scope.go:122] "RemoveContainer" containerID="46f2aec7abb6822e8a4ed558a8ab24714a90d249a646c50995bc653ca67063cc"
Oct 30 01:15:08.618816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.617406    5104 scope.go:122] "RemoveContainer" containerID="d12351b50178e67bbf1f908f44cefa3421672d8ba0fc60fc121f1d2a3e6308ad"
Oct 30 01:15:08.618816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.618530    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" podStartSLOduration=8.61851122 podStartE2EDuration="8.61851122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:08.579746401 +0000 UTC m=+1950.127436898" watchObservedRunningTime="2026-10-30 01:15:08.61851122 +0000 UTC m=+1950.166201711"
Oct 30 01:15:08.635370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.635342    5104 scope.go:122] "RemoveContainer" containerID="059e7b4c0ac1cc321ee3f925c722d9a014d7cb19a03252251234e9ea2726c72c"
Oct 30 01:15:08.884323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.884236    5104 generic.go:356] "Generic (PLEG): container finished" podID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerID="0c56ce3373e6c805a1033701ccf8fc7bfabb6227c23de6a590925555c854e2af" exitCode=60
Oct 30 01:15:08.884804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.884325    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" event={"ID":"78645a0a-feb8-41d3-95a2-addfb511c1fd","Type":"ContainerDied","Data":"0c56ce3373e6c805a1033701ccf8fc7bfabb6227c23de6a590925555c854e2af"}
Oct 30 01:15:08.884804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.884403    5104 scope.go:122] "RemoveContainer" containerID="ef0b8a25ab5dc812a21f711a3627ae12469cc991e6eb9865da7a909d484fa5e8"
Oct 30 01:15:08.884804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.884765    5104 scope.go:122] "RemoveContainer" containerID="0c56ce3373e6c805a1033701ccf8fc7bfabb6227c23de6a590925555c854e2af"
Oct 30 01:15:08.885343 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.885312    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5_odh-ai-gateway-infra(78645a0a-feb8-41d3-95a2-addfb511c1fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd"
Oct 30 01:15:08.886991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.886969    5104 generic.go:356] "Generic (PLEG): container finished" podID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerID="c8c0151c84a35a2a84895b0caf892a850793da9acc2f5f3f44ff93a216da0f50" exitCode=60
Oct 30 01:15:08.887104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.887029    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" event={"ID":"ffea74c7-864a-4a9f-b52a-2edd8aea5559","Type":"ContainerDied","Data":"c8c0151c84a35a2a84895b0caf892a850793da9acc2f5f3f44ff93a216da0f50"}
Oct 30 01:15:08.887539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.887519    5104 scope.go:122] "RemoveContainer" containerID="c8c0151c84a35a2a84895b0caf892a850793da9acc2f5f3f44ff93a216da0f50"
Oct 30 01:15:08.888004 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.887983    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn_odh-ai-gateway-infra(ffea74c7-864a-4a9f-b52a-2edd8aea5559)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559"
Oct 30 01:15:08.889398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.889374    5104 generic.go:356] "Generic (PLEG): container finished" podID="293a0b39-9772-4823-b91c-8f99316e92db" containerID="f28c40f9d7db2928b680b6f61acae80befe88c9259a9e5ff555a4406b4bebb9d" exitCode=60
Oct 30 01:15:08.889528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.889471    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerDied","Data":"f28c40f9d7db2928b680b6f61acae80befe88c9259a9e5ff555a4406b4bebb9d"}
Oct 30 01:15:08.889773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.889756    5104 scope.go:122] "RemoveContainer" containerID="f28c40f9d7db2928b680b6f61acae80befe88c9259a9e5ff555a4406b4bebb9d"
Oct 30 01:15:08.890203 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.890177    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m_odh-ai-gateway-infra(293a0b39-9772-4823-b91c-8f99316e92db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" podUID="293a0b39-9772-4823-b91c-8f99316e92db"
Oct 30 01:15:08.891802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.891781    5104 scope.go:122] "RemoveContainer" containerID="454cf7a8ecdb9542c11d92e460c355b29f9e76f91b10fa0c9ff6b70537fdf67d"
Oct 30 01:15:08.892191 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.892163    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5_odh-ai-gateway-infra(45a664ef-0a32-4a1a-98a1-5ccfdb4eed93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93"
Oct 30 01:15:08.894013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.893964    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerStarted","Data":"a53dcee9215339664ce79ad18a87dbb9da250ce285f7413268145fad90789078"}
Oct 30 01:15:08.895976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.895957    5104 scope.go:122] "RemoveContainer" containerID="20c3aab5ac3120a60a3daca5e8754ec4e9f02cb0a12fcb43c95a959e00c23796"
Oct 30 01:15:08.896259 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.896229    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv_odh-ai-gateway-infra(4cbf3f29-d8b1-4215-8c7d-b7311a0df231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231"
Oct 30 01:15:08.897679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.897655    5104 generic.go:356] "Generic (PLEG): container finished" podID="8a5910f9-37eb-42cd-a09d-417340f02866" containerID="018f9e289c5f236da869f077ab3c69a71089be525b57a839ff98e704dd34b05c" exitCode=60
Oct 30 01:15:08.897795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.897732    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" event={"ID":"8a5910f9-37eb-42cd-a09d-417340f02866","Type":"ContainerDied","Data":"018f9e289c5f236da869f077ab3c69a71089be525b57a839ff98e704dd34b05c"}
Oct 30 01:15:08.898026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.898008    5104 scope.go:122] "RemoveContainer" containerID="018f9e289c5f236da869f077ab3c69a71089be525b57a839ff98e704dd34b05c"
Oct 30 01:15:08.898373 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.898350    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw_odh-ai-gateway-infra(8a5910f9-37eb-42cd-a09d-417340f02866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" podUID="8a5910f9-37eb-42cd-a09d-417340f02866"
Oct 30 01:15:08.900085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.900069    5104 scope.go:122] "RemoveContainer" containerID="80638c9f9747572a90e8535612b84def5c89f4986fccafc550f3b29ba1c793c4"
Oct 30 01:15:08.900390 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.900361    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l_odh-ai-gateway-infra(5b931cfc-6a3d-4381-b5af-035ea30563cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb"
Oct 30 01:15:08.902076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.902059    5104 scope.go:122] "RemoveContainer" containerID="0f928e4a295a1096e3458debefe70f418db62944cdb852eb0bbd17ffd3aae748"
Oct 30 01:15:08.902318 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.902293    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg_odh-ai-gateway-infra(b1265940-ee90-4ede-9e4a-9bd690eb1681)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681"
Oct 30 01:15:08.902882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.902867    5104 scope.go:122] "RemoveContainer" containerID="407a87241c6407c04471227c43fc0f4a893fef54f785ac84b768ed416fdde0fc"
Oct 30 01:15:08.904152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.904036    5104 generic.go:356] "Generic (PLEG): container finished" podID="a510c363-24f2-41f1-869e-a509b345d5e5" containerID="66e4e9df2f7f21b9b9a6a0aed7e05012da63a146b8d56abf1cf8304e4c665a39" exitCode=60
Oct 30 01:15:08.904152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.904117    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" event={"ID":"a510c363-24f2-41f1-869e-a509b345d5e5","Type":"ContainerDied","Data":"66e4e9df2f7f21b9b9a6a0aed7e05012da63a146b8d56abf1cf8304e4c665a39"}
Oct 30 01:15:08.904357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.904337    5104 scope.go:122] "RemoveContainer" containerID="66e4e9df2f7f21b9b9a6a0aed7e05012da63a146b8d56abf1cf8304e4c665a39"
Oct 30 01:15:08.904652 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.904632    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq_odh-ai-gateway-infra(a510c363-24f2-41f1-869e-a509b345d5e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" podUID="a510c363-24f2-41f1-869e-a509b345d5e5"
Oct 30 01:15:08.907361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.907343    5104 scope.go:122] "RemoveContainer" containerID="9d2ab12ba29f814381c9dfb47dae47f15104f78d0f5e55f2e926f069a9ad3b6b"
Oct 30 01:15:08.907653 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.907632    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c_odh-ai-gateway-infra(1e6a9e4d-59c8-4aca-a777-c5f2510b1d16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16"
Oct 30 01:15:08.909079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.909052    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" event={"ID":"f0ae7c6c-84f1-47ab-8b4e-86a121949b97","Type":"ContainerStarted","Data":"df11e71a259e7c050632f1af49843a4bc99841e089ca1ac959c8879ebe648fbb"}
Oct 30 01:15:08.911173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.911155    5104 scope.go:122] "RemoveContainer" containerID="32f8d6cfba3993182622ccbb81ffa910bcdc344f15ca842bab26294686ae0c59"
Oct 30 01:15:08.911380 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:08.911363    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz_odh-ai-gateway-infra(25e08139-14a0-4499-aa79-6bb318c2f531)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" podUID="25e08139-14a0-4499-aa79-6bb318c2f531"
Oct 30 01:15:08.919312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.919290    5104 scope.go:122] "RemoveContainer" containerID="c265c26fffd2d97c80224f93ad1d2072805d8c0b9430270d398e88623238377b"
Oct 30 01:15:08.938545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.938519    5104 scope.go:122] "RemoveContainer" containerID="cde7154067a30b37b08a9277de7d25ddbfcac9f821b248ff30f1d936e3373448"
Oct 30 01:15:08.955691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:08.955661    5104 scope.go:122] "RemoveContainer" containerID="ce1b214d1fbef649a995bf00168e0fdcd2e3be4674cc359bf58c0fd80ad17ece"
Oct 30 01:15:09.109246 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.109212    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:15:09.923421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.923392    5104 scope.go:122] "RemoveContainer" containerID="0c56ce3373e6c805a1033701ccf8fc7bfabb6227c23de6a590925555c854e2af"
Oct 30 01:15:09.923921 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.923870    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5_odh-ai-gateway-infra(78645a0a-feb8-41d3-95a2-addfb511c1fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd"
Oct 30 01:15:09.925111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.925090    5104 scope.go:122] "RemoveContainer" containerID="c8c0151c84a35a2a84895b0caf892a850793da9acc2f5f3f44ff93a216da0f50"
Oct 30 01:15:09.925437 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.925410    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn_odh-ai-gateway-infra(ffea74c7-864a-4a9f-b52a-2edd8aea5559)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559"
Oct 30 01:15:09.926827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.926809    5104 scope.go:122] "RemoveContainer" containerID="f28c40f9d7db2928b680b6f61acae80befe88c9259a9e5ff555a4406b4bebb9d"
Oct 30 01:15:09.927140 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.927115    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m_odh-ai-gateway-infra(293a0b39-9772-4823-b91c-8f99316e92db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" podUID="293a0b39-9772-4823-b91c-8f99316e92db"
Oct 30 01:15:09.928619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.928595    5104 generic.go:356] "Generic (PLEG): container finished" podID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerID="a53dcee9215339664ce79ad18a87dbb9da250ce285f7413268145fad90789078" exitCode=60
Oct 30 01:15:09.928718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.928688    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerDied","Data":"a53dcee9215339664ce79ad18a87dbb9da250ce285f7413268145fad90789078"}
Oct 30 01:15:09.928783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.928744    5104 scope.go:122] "RemoveContainer" containerID="1c5d6dbd7e9237122ed385f07943c045178070f2a1793ecb8676d28393ebce74"
Oct 30 01:15:09.928902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.928887    5104 scope.go:122] "RemoveContainer" containerID="a53dcee9215339664ce79ad18a87dbb9da250ce285f7413268145fad90789078"
Oct 30 01:15:09.929222 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.929201    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs_odh-ai-gateway-infra(c6d2cd1c-ddd7-419a-8507-7e355abaf111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111"
Oct 30 01:15:09.930554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.930536    5104 scope.go:122] "RemoveContainer" containerID="018f9e289c5f236da869f077ab3c69a71089be525b57a839ff98e704dd34b05c"
Oct 30 01:15:09.930799 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.930777    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw_odh-ai-gateway-infra(8a5910f9-37eb-42cd-a09d-417340f02866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" podUID="8a5910f9-37eb-42cd-a09d-417340f02866"
Oct 30 01:15:09.932369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.932349    5104 scope.go:122] "RemoveContainer" containerID="66e4e9df2f7f21b9b9a6a0aed7e05012da63a146b8d56abf1cf8304e4c665a39"
Oct 30 01:15:09.932631 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.932609    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq_odh-ai-gateway-infra(a510c363-24f2-41f1-869e-a509b345d5e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" podUID="a510c363-24f2-41f1-869e-a509b345d5e5"
Oct 30 01:15:09.934244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.934221    5104 generic.go:356] "Generic (PLEG): container finished" podID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerID="df11e71a259e7c050632f1af49843a4bc99841e089ca1ac959c8879ebe648fbb" exitCode=60
Oct 30 01:15:09.934324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.934257    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" event={"ID":"f0ae7c6c-84f1-47ab-8b4e-86a121949b97","Type":"ContainerDied","Data":"df11e71a259e7c050632f1af49843a4bc99841e089ca1ac959c8879ebe648fbb"}
Oct 30 01:15:09.934557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.934541    5104 scope.go:122] "RemoveContainer" containerID="df11e71a259e7c050632f1af49843a4bc99841e089ca1ac959c8879ebe648fbb"
Oct 30 01:15:09.934870 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:09.934849    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb_odh-ai-gateway-infra(f0ae7c6c-84f1-47ab-8b4e-86a121949b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97"
Oct 30 01:15:09.944208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:09.944188    5104 scope.go:122] "RemoveContainer" containerID="9598076129e6d5ae27d6cd579b1990c4c2aae25572be94b12582ef7b51e8984a"
Oct 30 01:15:10.949572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:10.949541    5104 scope.go:122] "RemoveContainer" containerID="a53dcee9215339664ce79ad18a87dbb9da250ce285f7413268145fad90789078"
Oct 30 01:15:10.950083 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:10.950053    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs_odh-ai-gateway-infra(c6d2cd1c-ddd7-419a-8507-7e355abaf111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111"
Oct 30 01:15:10.951573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:10.951552    5104 scope.go:122] "RemoveContainer" containerID="df11e71a259e7c050632f1af49843a4bc99841e089ca1ac959c8879ebe648fbb"
Oct 30 01:15:10.951927 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:10.951905    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb_odh-ai-gateway-infra(f0ae7c6c-84f1-47ab-8b4e-86a121949b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97"
Oct 30 01:15:13.647507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.647469    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"]
Oct 30 01:15:13.654633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.654608    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.672718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.670920    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"]
Oct 30 01:15:13.680432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.672905    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50d078\"/\"kube-root-ca.crt\""
Oct 30 01:15:13.680432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.673670    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50d078\"/\"openshift-service-ca.crt\""
Oct 30 01:15:13.682914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.682291    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50d078\"/\"default-dockercfg-gzmqd\""
Oct 30 01:15:13.682914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.682291    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50d078\"/\"test-model-50d078-kserve-self-signed-certs\""
Oct 30 01:15:13.774050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.774005    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-home\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.774262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.774057    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tmp-dir\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.774262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.774160    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tls-certs\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.774426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.774397    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-dshm\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.774691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.774658    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-model-cache\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.774788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.774714    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ss5v\" (UniqueName: \"kubernetes.io/projected/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-kube-api-access-4ss5v\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.882426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.882297    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-model-cache\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.890059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.890028    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ss5v\" (UniqueName: \"kubernetes.io/projected/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-kube-api-access-4ss5v\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.902638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.902550    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-home\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.903229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.903206    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tmp-dir\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.903447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.903431    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tls-certs\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.903754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.903733    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-dshm\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.921545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.910573    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tmp-dir\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.921545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.889931    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-model-cache\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.921545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.903153    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-home\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.922644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.922611    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-dshm\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.955496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.937919    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tls-certs\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.962606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.956734    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ss5v\" (UniqueName: \"kubernetes.io/projected/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-kube-api-access-4ss5v\") pod \"test-model-50d078-kserve-568bcf4f8c-h49zg\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") " pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:13.986170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:13.982119    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:14.429497 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:14.429449    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a27a7b6_042c_4a01_a8a7_79afa3b7f400.slice/crio-a636181da5439fc2a4d454fa163533ec5b0aa063a5e7e34b60ae89c128dd9c8a WatchSource:0}: Error finding container a636181da5439fc2a4d454fa163533ec5b0aa063a5e7e34b60ae89c128dd9c8a: Status 404 returned error can't find the container with id a636181da5439fc2a4d454fa163533ec5b0aa063a5e7e34b60ae89c128dd9c8a
Oct 30 01:15:14.429896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:14.429870    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"]
Oct 30 01:15:15.190485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:15.188357    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg" event={"ID":"9a27a7b6-042c-4a01-a8a7-79afa3b7f400","Type":"ContainerStarted","Data":"2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54"}
Oct 30 01:15:15.190485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:15.188410    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg" event={"ID":"9a27a7b6-042c-4a01-a8a7-79afa3b7f400","Type":"ContainerStarted","Data":"a636181da5439fc2a4d454fa163533ec5b0aa063a5e7e34b60ae89c128dd9c8a"}
Oct 30 01:15:15.234214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:15.234163    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg" podStartSLOduration=2.234148812 podStartE2EDuration="2.234148812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:15.231869722 +0000 UTC m=+1956.779560221" watchObservedRunningTime="2026-10-30 01:15:15.234148812 +0000 UTC m=+1956.781839345"
Oct 30 01:15:15.983302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:15.983260    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:18.579704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:18.579661    5104 scope.go:122] "RemoveContainer" containerID="80638c9f9747572a90e8535612b84def5c89f4986fccafc550f3b29ba1c793c4"
Oct 30 01:15:18.580981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:18.580613    5104 scope.go:122] "RemoveContainer" containerID="0f928e4a295a1096e3458debefe70f418db62944cdb852eb0bbd17ffd3aae748"
Oct 30 01:15:18.580981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:18.580799    5104 scope.go:122] "RemoveContainer" containerID="251a46953ccfcb6942c5364d70fa9f636b166a1a014b92575e03f4df442b4249"
Oct 30 01:15:18.581133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:18.580977    5104 scope.go:122] "RemoveContainer" containerID="32f8d6cfba3993182622ccbb81ffa910bcdc344f15ca842bab26294686ae0c59"
Oct 30 01:15:18.582809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:18.581355    5104 scope.go:122] "RemoveContainer" containerID="20c3aab5ac3120a60a3daca5e8754ec4e9f02cb0a12fcb43c95a959e00c23796"
Oct 30 01:15:18.582809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:18.581450    5104 scope.go:122] "RemoveContainer" containerID="f28c40f9d7db2928b680b6f61acae80befe88c9259a9e5ff555a4406b4bebb9d"
Oct 30 01:15:19.250134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:19.250085    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerStarted","Data":"f7ed1e5f5c1d4b7fb0c3014e4b4c1b0d555a7f63bee3a3ef9a98dda0ac887b07"}
Oct 30 01:15:19.252714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:19.252681    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerStarted","Data":"ecbedcb3672892abdb7c98042bb0b206c8ae24c8cf9f4354f63c5465afd1e601"}
Oct 30 01:15:19.570123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:19.570032    5104 scope.go:122] "RemoveContainer" containerID="9fc49d66a8d02a0b3bc9bc5a1f7b43eff6faae606d2b45d53997f761c32b97d9"
Oct 30 01:15:19.570298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:19.570170    5104 scope.go:122] "RemoveContainer" containerID="0c56ce3373e6c805a1033701ccf8fc7bfabb6227c23de6a590925555c854e2af"
Oct 30 01:15:19.570298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:19.570273    5104 scope.go:122] "RemoveContainer" containerID="c8c0151c84a35a2a84895b0caf892a850793da9acc2f5f3f44ff93a216da0f50"
Oct 30 01:15:19.570404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:19.570386    5104 scope.go:122] "RemoveContainer" containerID="551cfe6d635fe456d2b08334f60d95243bcc519f246783f5145726da12ab32d5"
Oct 30 01:15:19.570963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:19.570907    5104 scope.go:122] "RemoveContainer" containerID="9d2ab12ba29f814381c9dfb47dae47f15104f78d0f5e55f2e926f069a9ad3b6b"
Oct 30 01:15:20.277680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.277407    5104 generic.go:356] "Generic (PLEG): container finished" podID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerID="f7ed1e5f5c1d4b7fb0c3014e4b4c1b0d555a7f63bee3a3ef9a98dda0ac887b07" exitCode=60
Oct 30 01:15:20.277680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.277615    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerDied","Data":"f7ed1e5f5c1d4b7fb0c3014e4b4c1b0d555a7f63bee3a3ef9a98dda0ac887b07"}
Oct 30 01:15:20.279444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.277760    5104 scope.go:122] "RemoveContainer" containerID="0f928e4a295a1096e3458debefe70f418db62944cdb852eb0bbd17ffd3aae748"
Oct 30 01:15:20.279444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.278196    5104 scope.go:122] "RemoveContainer" containerID="f7ed1e5f5c1d4b7fb0c3014e4b4c1b0d555a7f63bee3a3ef9a98dda0ac887b07"
Oct 30 01:15:20.279444 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:20.278675    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg_odh-ai-gateway-infra(b1265940-ee90-4ede-9e4a-9bd690eb1681)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681"
Oct 30 01:15:20.285047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.285024    5104 generic.go:356] "Generic (PLEG): container finished" podID="c0f66867-6e46-479b-a883-33b036fa6279" containerID="a6494772447e460c4f160abbc1d1f029f21f2996d4b0e2287f807e0b0c0cbe1e" exitCode=60
Oct 30 01:15:20.285166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.285096    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" event={"ID":"c0f66867-6e46-479b-a883-33b036fa6279","Type":"ContainerDied","Data":"a6494772447e460c4f160abbc1d1f029f21f2996d4b0e2287f807e0b0c0cbe1e"}
Oct 30 01:15:20.285711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.285687    5104 scope.go:122] "RemoveContainer" containerID="a6494772447e460c4f160abbc1d1f029f21f2996d4b0e2287f807e0b0c0cbe1e"
Oct 30 01:15:20.286219 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:20.286192    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97_odh-ai-gateway-infra(c0f66867-6e46-479b-a883-33b036fa6279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" podUID="c0f66867-6e46-479b-a883-33b036fa6279"
Oct 30 01:15:20.289386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.289354    5104 generic.go:356] "Generic (PLEG): container finished" podID="25e08139-14a0-4499-aa79-6bb318c2f531" containerID="5e8fcfc0d63c8d6aec09a47cd602531121ae9d6519b3d97e6b9d96d641fdd0cd" exitCode=60
Oct 30 01:15:20.289543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.289450    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" event={"ID":"25e08139-14a0-4499-aa79-6bb318c2f531","Type":"ContainerDied","Data":"5e8fcfc0d63c8d6aec09a47cd602531121ae9d6519b3d97e6b9d96d641fdd0cd"}
Oct 30 01:15:20.289789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.289774    5104 scope.go:122] "RemoveContainer" containerID="5e8fcfc0d63c8d6aec09a47cd602531121ae9d6519b3d97e6b9d96d641fdd0cd"
Oct 30 01:15:20.290040 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:20.290018    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz_odh-ai-gateway-infra(25e08139-14a0-4499-aa79-6bb318c2f531)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" podUID="25e08139-14a0-4499-aa79-6bb318c2f531"
Oct 30 01:15:20.293433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.292438    5104 generic.go:356] "Generic (PLEG): container finished" podID="293a0b39-9772-4823-b91c-8f99316e92db" containerID="ecbedcb3672892abdb7c98042bb0b206c8ae24c8cf9f4354f63c5465afd1e601" exitCode=60
Oct 30 01:15:20.293433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.292654    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerDied","Data":"ecbedcb3672892abdb7c98042bb0b206c8ae24c8cf9f4354f63c5465afd1e601"}
Oct 30 01:15:20.293433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.293040    5104 scope.go:122] "RemoveContainer" containerID="ecbedcb3672892abdb7c98042bb0b206c8ae24c8cf9f4354f63c5465afd1e601"
Oct 30 01:15:20.293433 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:20.293359    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m_odh-ai-gateway-infra(293a0b39-9772-4823-b91c-8f99316e92db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" podUID="293a0b39-9772-4823-b91c-8f99316e92db"
Oct 30 01:15:20.295892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.295869    5104 generic.go:356] "Generic (PLEG): container finished" podID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerID="1a94f232798a82afaf8bd6d7d5c868030ef31de29d1ef95690b3ac2d83f6703a" exitCode=60
Oct 30 01:15:20.295990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.295963    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" event={"ID":"4cbf3f29-d8b1-4215-8c7d-b7311a0df231","Type":"ContainerDied","Data":"1a94f232798a82afaf8bd6d7d5c868030ef31de29d1ef95690b3ac2d83f6703a"}
Oct 30 01:15:20.296347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.296328    5104 scope.go:122] "RemoveContainer" containerID="1a94f232798a82afaf8bd6d7d5c868030ef31de29d1ef95690b3ac2d83f6703a"
Oct 30 01:15:20.296662 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:20.296638    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv_odh-ai-gateway-infra(4cbf3f29-d8b1-4215-8c7d-b7311a0df231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231"
Oct 30 01:15:20.299054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.299032    5104 generic.go:356] "Generic (PLEG): container finished" podID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerID="94d8693b9bca24dc5c457a18aad52a00d91306c17dc8e6dc580b3cbb5f2f3a97" exitCode=60
Oct 30 01:15:20.299157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.299077    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" event={"ID":"5b931cfc-6a3d-4381-b5af-035ea30563cb","Type":"ContainerDied","Data":"94d8693b9bca24dc5c457a18aad52a00d91306c17dc8e6dc580b3cbb5f2f3a97"}
Oct 30 01:15:20.299429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.299412    5104 scope.go:122] "RemoveContainer" containerID="94d8693b9bca24dc5c457a18aad52a00d91306c17dc8e6dc580b3cbb5f2f3a97"
Oct 30 01:15:20.299781 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:20.299758    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l_odh-ai-gateway-infra(5b931cfc-6a3d-4381-b5af-035ea30563cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb"
Oct 30 01:15:20.302086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.302069    5104 scope.go:122] "RemoveContainer" containerID="251a46953ccfcb6942c5364d70fa9f636b166a1a014b92575e03f4df442b4249"
Oct 30 01:15:20.320128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.320108    5104 scope.go:122] "RemoveContainer" containerID="32f8d6cfba3993182622ccbb81ffa910bcdc344f15ca842bab26294686ae0c59"
Oct 30 01:15:20.336357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.336340    5104 scope.go:122] "RemoveContainer" containerID="f28c40f9d7db2928b680b6f61acae80befe88c9259a9e5ff555a4406b4bebb9d"
Oct 30 01:15:20.336995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.336968    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"]
Oct 30 01:15:20.352576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.352552    5104 scope.go:122] "RemoveContainer" containerID="20c3aab5ac3120a60a3daca5e8754ec4e9f02cb0a12fcb43c95a959e00c23796"
Oct 30 01:15:20.358844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.358816    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"]
Oct 30 01:15:20.375669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.375644    5104 scope.go:122] "RemoveContainer" containerID="80638c9f9747572a90e8535612b84def5c89f4986fccafc550f3b29ba1c793c4"
Oct 30 01:15:20.572072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.572044    5104 scope.go:122] "RemoveContainer" containerID="a53dcee9215339664ce79ad18a87dbb9da250ce285f7413268145fad90789078"
Oct 30 01:15:20.575928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.575717    5104 scope.go:122] "RemoveContainer" containerID="66e4e9df2f7f21b9b9a6a0aed7e05012da63a146b8d56abf1cf8304e4c665a39"
Oct 30 01:15:20.581266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.577755    5104 scope.go:122] "RemoveContainer" containerID="454cf7a8ecdb9542c11d92e460c355b29f9e76f91b10fa0c9ff6b70537fdf67d"
Oct 30 01:15:20.581266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.578344    5104 scope.go:122] "RemoveContainer" containerID="018f9e289c5f236da869f077ab3c69a71089be525b57a839ff98e704dd34b05c"
Oct 30 01:15:20.581266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.579129    5104 scope.go:122] "RemoveContainer" containerID="568872b502a78c844b9ebfc81c40c44c4620de04a2b34d97e936956171a36e03"
Oct 30 01:15:20.581734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:20.581360    5104 scope.go:122] "RemoveContainer" containerID="8d9b550b636c4019951ee9f13b9857f2c041193e4845edfc646891ce2d71f486"
Oct 30 01:15:21.315811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.315779    5104 generic.go:356] "Generic (PLEG): container finished" podID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerID="7091cd5cded569e893258e9a0b6efcec00d69b55d507b6991440f10cacb8a971" exitCode=60
Oct 30 01:15:21.316214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.315852    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" event={"ID":"78645a0a-feb8-41d3-95a2-addfb511c1fd","Type":"ContainerDied","Data":"7091cd5cded569e893258e9a0b6efcec00d69b55d507b6991440f10cacb8a971"}
Oct 30 01:15:21.316214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.315926    5104 scope.go:122] "RemoveContainer" containerID="0c56ce3373e6c805a1033701ccf8fc7bfabb6227c23de6a590925555c854e2af"
Oct 30 01:15:21.316465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.316435    5104 scope.go:122] "RemoveContainer" containerID="7091cd5cded569e893258e9a0b6efcec00d69b55d507b6991440f10cacb8a971"
Oct 30 01:15:21.316956 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:21.316925    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5_odh-ai-gateway-infra(78645a0a-feb8-41d3-95a2-addfb511c1fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd"
Oct 30 01:15:21.319974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.319954    5104 generic.go:356] "Generic (PLEG): container finished" podID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerID="21cef5df943a8ba41007a84605a15bbb066923b0a08ce9d1839311d18bcf8e66" exitCode=60
Oct 30 01:15:21.320107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.320033    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" event={"ID":"ffea74c7-864a-4a9f-b52a-2edd8aea5559","Type":"ContainerDied","Data":"21cef5df943a8ba41007a84605a15bbb066923b0a08ce9d1839311d18bcf8e66"}
Oct 30 01:15:21.320553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.320535    5104 scope.go:122] "RemoveContainer" containerID="21cef5df943a8ba41007a84605a15bbb066923b0a08ce9d1839311d18bcf8e66"
Oct 30 01:15:21.320928 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:21.320906    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn_odh-ai-gateway-infra(ffea74c7-864a-4a9f-b52a-2edd8aea5559)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559"
Oct 30 01:15:21.324650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.324623    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerStarted","Data":"fac52f23e4b262d1eccc5a1ea9d3f6e8e216c134e9604ca1d7cdd623704e8eb9"}
Oct 30 01:15:21.332811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.332784    5104 generic.go:356] "Generic (PLEG): container finished" podID="0489e1db-1b58-4955-966b-82a17015a4be" containerID="b756b6ee6ab904abe6ebe7f93864096a2211addad011225ae4dcd880b34f1f14" exitCode=60
Oct 30 01:15:21.332941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.332923    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" event={"ID":"0489e1db-1b58-4955-966b-82a17015a4be","Type":"ContainerDied","Data":"b756b6ee6ab904abe6ebe7f93864096a2211addad011225ae4dcd880b34f1f14"}
Oct 30 01:15:21.333410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.333387    5104 scope.go:122] "RemoveContainer" containerID="b756b6ee6ab904abe6ebe7f93864096a2211addad011225ae4dcd880b34f1f14"
Oct 30 01:15:21.333859 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:21.333836    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29_odh-ai-gateway-infra(0489e1db-1b58-4955-966b-82a17015a4be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" podUID="0489e1db-1b58-4955-966b-82a17015a4be"
Oct 30 01:15:21.338101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.338078    5104 generic.go:356] "Generic (PLEG): container finished" podID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerID="c4ed081ef8fb9258f44c8699820d3f562d092fe6bf1555e4f2cecf6d56c72cef" exitCode=60
Oct 30 01:15:21.338233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.338102    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" event={"ID":"ee09f6bd-6a43-4113-9d5c-84029ec53140","Type":"ContainerDied","Data":"c4ed081ef8fb9258f44c8699820d3f562d092fe6bf1555e4f2cecf6d56c72cef"}
Oct 30 01:15:21.338617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.338597    5104 scope.go:122] "RemoveContainer" containerID="c4ed081ef8fb9258f44c8699820d3f562d092fe6bf1555e4f2cecf6d56c72cef"
Oct 30 01:15:21.339052 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:21.339015    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx_odh-ai-gateway-infra(ee09f6bd-6a43-4113-9d5c-84029ec53140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140"
Oct 30 01:15:21.340527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.340503    5104 generic.go:356] "Generic (PLEG): container finished" podID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerID="c641bde82069dda1c63cdc5441b1b90ff82a413b4a47eee793d5ceecfa030faa" exitCode=60
Oct 30 01:15:21.340643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.340552    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" event={"ID":"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16","Type":"ContainerDied","Data":"c641bde82069dda1c63cdc5441b1b90ff82a413b4a47eee793d5ceecfa030faa"}
Oct 30 01:15:21.341038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.341015    5104 scope.go:122] "RemoveContainer" containerID="c641bde82069dda1c63cdc5441b1b90ff82a413b4a47eee793d5ceecfa030faa"
Oct 30 01:15:21.341498 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:21.341469    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c_odh-ai-gateway-infra(1e6a9e4d-59c8-4aca-a777-c5f2510b1d16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16"
Oct 30 01:15:21.365352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.365323    5104 scope.go:122] "RemoveContainer" containerID="c8c0151c84a35a2a84895b0caf892a850793da9acc2f5f3f44ff93a216da0f50"
Oct 30 01:15:21.403757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.402523    5104 scope.go:122] "RemoveContainer" containerID="551cfe6d635fe456d2b08334f60d95243bcc519f246783f5145726da12ab32d5"
Oct 30 01:15:21.427646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.427621    5104 scope.go:122] "RemoveContainer" containerID="9fc49d66a8d02a0b3bc9bc5a1f7b43eff6faae606d2b45d53997f761c32b97d9"
Oct 30 01:15:21.440184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.439843    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"]
Oct 30 01:15:21.517842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.513142    5104 scope.go:122] "RemoveContainer" containerID="9d2ab12ba29f814381c9dfb47dae47f15104f78d0f5e55f2e926f069a9ad3b6b"
Oct 30 01:15:21.576180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.576152    5104 scope.go:122] "RemoveContainer" containerID="df11e71a259e7c050632f1af49843a4bc99841e089ca1ac959c8879ebe648fbb"
Oct 30 01:15:21.635237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.634948    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"]
Oct 30 01:15:21.649808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.640777    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:21.654094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.654058    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"]
Oct 30 01:15:21.709193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.701677    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"]
Oct 30 01:15:21.741587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.740038    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1265940-ee90-4ede-9e4a-9bd690eb1681-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1265940-ee90-4ede-9e4a-9bd690eb1681-maas-api-service-ca\") pod \"b1265940-ee90-4ede-9e4a-9bd690eb1681\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") "
Oct 30 01:15:21.741587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.740371    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1265940-ee90-4ede-9e4a-9bd690eb1681-kube-api-access-5qbg2\" (UniqueName: \"kubernetes.io/projected/b1265940-ee90-4ede-9e4a-9bd690eb1681-kube-api-access-5qbg2\") pod \"b1265940-ee90-4ede-9e4a-9bd690eb1681\" (UID: \"b1265940-ee90-4ede-9e4a-9bd690eb1681\") "
Oct 30 01:15:21.742440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.742413    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1265940-ee90-4ede-9e4a-9bd690eb1681-maas-api-service-ca" pod "b1265940-ee90-4ede-9e4a-9bd690eb1681" (UID: "b1265940-ee90-4ede-9e4a-9bd690eb1681"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:21.834558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.834396    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:21.849351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.849304    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1265940-ee90-4ede-9e4a-9bd690eb1681-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.909160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.908890    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1265940-ee90-4ede-9e4a-9bd690eb1681-kube-api-access-5qbg2" pod "b1265940-ee90-4ede-9e4a-9bd690eb1681" (UID: "b1265940-ee90-4ede-9e4a-9bd690eb1681"). InnerVolumeSpecName "kube-api-access-5qbg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.956496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.955068    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293a0b39-9772-4823-b91c-8f99316e92db-kube-api-access-ksppq\" (UniqueName: \"kubernetes.io/projected/293a0b39-9772-4823-b91c-8f99316e92db-kube-api-access-ksppq\") pod \"293a0b39-9772-4823-b91c-8f99316e92db\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") "
Oct 30 01:15:21.956496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.955371    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293a0b39-9772-4823-b91c-8f99316e92db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293a0b39-9772-4823-b91c-8f99316e92db-maas-api-service-ca\") pod \"293a0b39-9772-4823-b91c-8f99316e92db\" (UID: \"293a0b39-9772-4823-b91c-8f99316e92db\") "
Oct 30 01:15:21.967514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.958625    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbg2\" (UniqueName: \"kubernetes.io/projected/b1265940-ee90-4ede-9e4a-9bd690eb1681-kube-api-access-5qbg2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.973944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.973889    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293a0b39-9772-4823-b91c-8f99316e92db-kube-api-access-ksppq" pod "293a0b39-9772-4823-b91c-8f99316e92db" (UID: "293a0b39-9772-4823-b91c-8f99316e92db"). InnerVolumeSpecName "kube-api-access-ksppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.983811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:21.977783    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293a0b39-9772-4823-b91c-8f99316e92db-maas-api-service-ca" pod "293a0b39-9772-4823-b91c-8f99316e92db" (UID: "293a0b39-9772-4823-b91c-8f99316e92db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.067391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.067345    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293a0b39-9772-4823-b91c-8f99316e92db-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.067391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.067380    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksppq\" (UniqueName: \"kubernetes.io/projected/293a0b39-9772-4823-b91c-8f99316e92db-kube-api-access-ksppq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.136916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.136167    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:22.170805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.170776    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-maas-api-service-ca\") pod \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") "
Oct 30 01:15:22.171326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.171302    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-kube-api-access-599qq\" (UniqueName: \"kubernetes.io/projected/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-kube-api-access-599qq\") pod \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\" (UID: \"4cbf3f29-d8b1-4215-8c7d-b7311a0df231\") "
Oct 30 01:15:22.171569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.171380    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-maas-api-service-ca" pod "4cbf3f29-d8b1-4215-8c7d-b7311a0df231" (UID: "4cbf3f29-d8b1-4215-8c7d-b7311a0df231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.176603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.175923    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-kube-api-access-599qq" pod "4cbf3f29-d8b1-4215-8c7d-b7311a0df231" (UID: "4cbf3f29-d8b1-4215-8c7d-b7311a0df231"). InnerVolumeSpecName "kube-api-access-599qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.193381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.188983    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.193381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.189016    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599qq\" (UniqueName: \"kubernetes.io/projected/4cbf3f29-d8b1-4215-8c7d-b7311a0df231-kube-api-access-599qq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.333169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.333143    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:22.337592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.337572    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:22.342811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.342790    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:22.351049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.351021    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97" event={"ID":"c0f66867-6e46-479b-a883-33b036fa6279","Type":"ContainerDied","Data":"c121382d1c80367f27db3ac49d82fd8c5c784d785c67a46e9c320ab15a241707"}
Oct 30 01:15:22.352843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.351071    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"
Oct 30 01:15:22.352843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.351092    5104 scope.go:122] "RemoveContainer" containerID="a6494772447e460c4f160abbc1d1f029f21f2996d4b0e2287f807e0b0c0cbe1e"
Oct 30 01:15:22.354172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.354141    5104 generic.go:356] "Generic (PLEG): container finished" podID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerID="85fdb21c581fbd57cc18e5ecd8ae3595145c109934de23e9ee61acb53e2edd83" exitCode=60
Oct 30 01:15:22.354308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.354212    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" event={"ID":"8b270123-f6a5-4202-b8e3-699b84d55ee9","Type":"ContainerDied","Data":"85fdb21c581fbd57cc18e5ecd8ae3595145c109934de23e9ee61acb53e2edd83"}
Oct 30 01:15:22.354749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.354731    5104 scope.go:122] "RemoveContainer" containerID="85fdb21c581fbd57cc18e5ecd8ae3595145c109934de23e9ee61acb53e2edd83"
Oct 30 01:15:22.355206 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:22.355176    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7_odh-ai-gateway-infra(8b270123-f6a5-4202-b8e3-699b84d55ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9"
Oct 30 01:15:22.356526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.356505    5104 generic.go:356] "Generic (PLEG): container finished" podID="a510c363-24f2-41f1-869e-a509b345d5e5" containerID="4c4ed5f6ec89d10a7f3371883e4341c920033a186265a162dedb23249b8cfbf4" exitCode=60
Oct 30 01:15:22.356685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.356659    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" event={"ID":"a510c363-24f2-41f1-869e-a509b345d5e5","Type":"ContainerDied","Data":"4c4ed5f6ec89d10a7f3371883e4341c920033a186265a162dedb23249b8cfbf4"}
Oct 30 01:15:22.357160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.357142    5104 scope.go:122] "RemoveContainer" containerID="4c4ed5f6ec89d10a7f3371883e4341c920033a186265a162dedb23249b8cfbf4"
Oct 30 01:15:22.362723 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:22.362657    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq_odh-ai-gateway-infra(a510c363-24f2-41f1-869e-a509b345d5e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" podUID="a510c363-24f2-41f1-869e-a509b345d5e5"
Oct 30 01:15:22.363914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.363890    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"
Oct 30 01:15:22.364014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.363914    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz" event={"ID":"25e08139-14a0-4499-aa79-6bb318c2f531","Type":"ContainerDied","Data":"e7475e95da47d5fd6e25d6f2f3ba1520d00feecb62d71479213fd95fcf286db9"}
Oct 30 01:15:22.367532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.367442    5104 scope.go:122] "RemoveContainer" containerID="8d9b550b636c4019951ee9f13b9857f2c041193e4845edfc646891ce2d71f486"
Oct 30 01:15:22.367763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.367742    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"]
Oct 30 01:15:22.370566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.370530    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m" event={"ID":"293a0b39-9772-4823-b91c-8f99316e92db","Type":"ContainerDied","Data":"4733e114d5a3ed4fa50a0568812af48f1f5e277ecc7a01c838c3629acd3f51d9"}
Oct 30 01:15:22.370654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.370547    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"
Oct 30 01:15:22.381441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.379778    5104 generic.go:356] "Generic (PLEG): container finished" podID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerID="b63d4be038f65dc8aaed0788e2bd0c5de83f32ef26929a50d2d602cc60be2a4f" exitCode=60
Oct 30 01:15:22.381441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.379888    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" event={"ID":"aad6b4c4-517f-41a2-b986-e7367bef493b","Type":"ContainerDied","Data":"b63d4be038f65dc8aaed0788e2bd0c5de83f32ef26929a50d2d602cc60be2a4f"}
Oct 30 01:15:22.381441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.380409    5104 scope.go:122] "RemoveContainer" containerID="b63d4be038f65dc8aaed0788e2bd0c5de83f32ef26929a50d2d602cc60be2a4f"
Oct 30 01:15:22.381441 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:22.380926    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l_odh-ai-gateway-infra(aad6b4c4-517f-41a2-b986-e7367bef493b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b"
Oct 30 01:15:22.386072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.384926    5104 generic.go:356] "Generic (PLEG): container finished" podID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerID="724c12226aaecceaa1589d4aecdc490e9ff41f87d61d52452182fcd15c003265" exitCode=60
Oct 30 01:15:22.386072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.385077    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" event={"ID":"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93","Type":"ContainerDied","Data":"724c12226aaecceaa1589d4aecdc490e9ff41f87d61d52452182fcd15c003265"}
Oct 30 01:15:22.386072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.385601    5104 scope.go:122] "RemoveContainer" containerID="724c12226aaecceaa1589d4aecdc490e9ff41f87d61d52452182fcd15c003265"
Oct 30 01:15:22.386727 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:22.386143    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5_odh-ai-gateway-infra(45a664ef-0a32-4a1a-98a1-5ccfdb4eed93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93"
Oct 30 01:15:22.394165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.389166    5104 generic.go:356] "Generic (PLEG): container finished" podID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerID="fac52f23e4b262d1eccc5a1ea9d3f6e8e216c134e9604ca1d7cdd623704e8eb9" exitCode=60
Oct 30 01:15:22.394165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.389245    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerDied","Data":"fac52f23e4b262d1eccc5a1ea9d3f6e8e216c134e9604ca1d7cdd623704e8eb9"}
Oct 30 01:15:22.394165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.392560    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b931cfc-6a3d-4381-b5af-035ea30563cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b931cfc-6a3d-4381-b5af-035ea30563cb-maas-api-service-ca\") pod \"5b931cfc-6a3d-4381-b5af-035ea30563cb\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") "
Oct 30 01:15:22.394165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.392629    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e08139-14a0-4499-aa79-6bb318c2f531-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e08139-14a0-4499-aa79-6bb318c2f531-maas-api-service-ca\") pod \"25e08139-14a0-4499-aa79-6bb318c2f531\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") "
Oct 30 01:15:22.394165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.393424    5104 scope.go:122] "RemoveContainer" containerID="fac52f23e4b262d1eccc5a1ea9d3f6e8e216c134e9604ca1d7cdd623704e8eb9"
Oct 30 01:15:22.394165 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:22.393917    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs_odh-ai-gateway-infra(c6d2cd1c-ddd7-419a-8507-7e355abaf111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111"
Oct 30 01:15:22.394165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.394053    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv" event={"ID":"4cbf3f29-d8b1-4215-8c7d-b7311a0df231","Type":"ContainerDied","Data":"80dcd7078d44ba93d2524023b80d7a240e0a0434fbd1511927c83cf1e5318b86"}
Oct 30 01:15:22.394640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.394571    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"
Oct 30 01:15:22.397546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.395078    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e08139-14a0-4499-aa79-6bb318c2f531-maas-api-service-ca" pod "25e08139-14a0-4499-aa79-6bb318c2f531" (UID: "25e08139-14a0-4499-aa79-6bb318c2f531"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.397546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.396711    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b931cfc-6a3d-4381-b5af-035ea30563cb-kube-api-access-r6xf4\" (UniqueName: \"kubernetes.io/projected/5b931cfc-6a3d-4381-b5af-035ea30563cb-kube-api-access-r6xf4\") pod \"5b931cfc-6a3d-4381-b5af-035ea30563cb\" (UID: \"5b931cfc-6a3d-4381-b5af-035ea30563cb\") "
Oct 30 01:15:22.397546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.396766    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f66867-6e46-479b-a883-33b036fa6279-kube-api-access-j59pn\" (UniqueName: \"kubernetes.io/projected/c0f66867-6e46-479b-a883-33b036fa6279-kube-api-access-j59pn\") pod \"c0f66867-6e46-479b-a883-33b036fa6279\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") "
Oct 30 01:15:22.397546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.396858    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f66867-6e46-479b-a883-33b036fa6279-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f66867-6e46-479b-a883-33b036fa6279-maas-api-service-ca\") pod \"c0f66867-6e46-479b-a883-33b036fa6279\" (UID: \"c0f66867-6e46-479b-a883-33b036fa6279\") "
Oct 30 01:15:22.397546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.396921    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e08139-14a0-4499-aa79-6bb318c2f531-kube-api-access-h26rs\" (UniqueName: \"kubernetes.io/projected/25e08139-14a0-4499-aa79-6bb318c2f531-kube-api-access-h26rs\") pod \"25e08139-14a0-4499-aa79-6bb318c2f531\" (UID: \"25e08139-14a0-4499-aa79-6bb318c2f531\") "
Oct 30 01:15:22.397895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.397674    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e08139-14a0-4499-aa79-6bb318c2f531-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.398678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.398095    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b931cfc-6a3d-4381-b5af-035ea30563cb-maas-api-service-ca" pod "5b931cfc-6a3d-4381-b5af-035ea30563cb" (UID: "5b931cfc-6a3d-4381-b5af-035ea30563cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.399158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.398935    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f66867-6e46-479b-a883-33b036fa6279-maas-api-service-ca" pod "c0f66867-6e46-479b-a883-33b036fa6279" (UID: "c0f66867-6e46-479b-a883-33b036fa6279"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.410492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.407795    5104 generic.go:356] "Generic (PLEG): container finished" podID="8a5910f9-37eb-42cd-a09d-417340f02866" containerID="6d03e0e714f1a7732aef6ca33939c669f7324f938127fde7ad5b3febe97c2c62" exitCode=60
Oct 30 01:15:22.410492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.407892    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" event={"ID":"8a5910f9-37eb-42cd-a09d-417340f02866","Type":"ContainerDied","Data":"6d03e0e714f1a7732aef6ca33939c669f7324f938127fde7ad5b3febe97c2c62"}
Oct 30 01:15:22.410492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.408422    5104 scope.go:122] "RemoveContainer" containerID="6d03e0e714f1a7732aef6ca33939c669f7324f938127fde7ad5b3febe97c2c62"
Oct 30 01:15:22.410492 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:22.408957    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw_odh-ai-gateway-infra(8a5910f9-37eb-42cd-a09d-417340f02866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" podUID="8a5910f9-37eb-42cd-a09d-417340f02866"
Oct 30 01:15:22.410492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.410012    5104 scope.go:122] "RemoveContainer" containerID="66e4e9df2f7f21b9b9a6a0aed7e05012da63a146b8d56abf1cf8304e4c665a39"
Oct 30 01:15:22.426419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.424268    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l" event={"ID":"5b931cfc-6a3d-4381-b5af-035ea30563cb","Type":"ContainerDied","Data":"2137778dc7d3b203f8702762a6bb9ac32c156e16ac126aac4a58d51f2ed44017"}
Oct 30 01:15:22.426419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.424393    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"
Oct 30 01:15:22.433682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.433656    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"]
Oct 30 01:15:22.455782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.455738    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg" event={"ID":"b1265940-ee90-4ede-9e4a-9bd690eb1681","Type":"ContainerDied","Data":"9512caaa1a925ae92a4d6be0eaffd2b1fe88624c9be8c5fb449ded71d78419e2"}
Oct 30 01:15:22.456490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.456421    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"
Oct 30 01:15:22.486770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.483171    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f66867-6e46-479b-a883-33b036fa6279-kube-api-access-j59pn" pod "c0f66867-6e46-479b-a883-33b036fa6279" (UID: "c0f66867-6e46-479b-a883-33b036fa6279"). InnerVolumeSpecName "kube-api-access-j59pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.486770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.486660    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e08139-14a0-4499-aa79-6bb318c2f531-kube-api-access-h26rs" pod "25e08139-14a0-4499-aa79-6bb318c2f531" (UID: "25e08139-14a0-4499-aa79-6bb318c2f531"). InnerVolumeSpecName "kube-api-access-h26rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.496489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.494277    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b931cfc-6a3d-4381-b5af-035ea30563cb-kube-api-access-r6xf4" pod "5b931cfc-6a3d-4381-b5af-035ea30563cb" (UID: "5b931cfc-6a3d-4381-b5af-035ea30563cb"). InnerVolumeSpecName "kube-api-access-r6xf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.512297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.512256    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"]
Oct 30 01:15:22.514841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.514449    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6xf4\" (UniqueName: \"kubernetes.io/projected/5b931cfc-6a3d-4381-b5af-035ea30563cb-kube-api-access-r6xf4\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.514841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.514495    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59pn\" (UniqueName: \"kubernetes.io/projected/c0f66867-6e46-479b-a883-33b036fa6279-kube-api-access-j59pn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.514841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.514511    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f66867-6e46-479b-a883-33b036fa6279-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.514841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.514525    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h26rs\" (UniqueName: \"kubernetes.io/projected/25e08139-14a0-4499-aa79-6bb318c2f531-kube-api-access-h26rs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.514841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.514541    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b931cfc-6a3d-4381-b5af-035ea30563cb-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.516343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.516322    5104 scope.go:122] "RemoveContainer" containerID="5e8fcfc0d63c8d6aec09a47cd602531121ae9d6519b3d97e6b9d96d641fdd0cd"
Oct 30 01:15:22.525636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.525613    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"]
Oct 30 01:15:22.543908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.543799    5104 scope.go:122] "RemoveContainer" containerID="ecbedcb3672892abdb7c98042bb0b206c8ae24c8cf9f4354f63c5465afd1e601"
Oct 30 01:15:22.576573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.568782    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"]
Oct 30 01:15:22.639650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.639615    5104 scope.go:122] "RemoveContainer" containerID="568872b502a78c844b9ebfc81c40c44c4620de04a2b34d97e936956171a36e03"
Oct 30 01:15:22.747927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.747677    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"]
Oct 30 01:15:22.766517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.765213    5104 scope.go:122] "RemoveContainer" containerID="454cf7a8ecdb9542c11d92e460c355b29f9e76f91b10fa0c9ff6b70537fdf67d"
Oct 30 01:15:22.771806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.771762    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:22.838473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.838401    5104 scope.go:122] "RemoveContainer" containerID="a53dcee9215339664ce79ad18a87dbb9da250ce285f7413268145fad90789078"
Oct 30 01:15:22.842034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.842002    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:22.913427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.912770    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"]
Oct 30 01:15:22.921253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.920126    5104 scope.go:122] "RemoveContainer" containerID="1a94f232798a82afaf8bd6d7d5c868030ef31de29d1ef95690b3ac2d83f6703a"
Oct 30 01:15:22.924541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.924514    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888715-sdv97"]
Oct 30 01:15:22.950495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.942348    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee09f6bd-6a43-4113-9d5c-84029ec53140-kube-api-access-h5zd9\" (UniqueName: \"kubernetes.io/projected/ee09f6bd-6a43-4113-9d5c-84029ec53140-kube-api-access-h5zd9\") pod \"ee09f6bd-6a43-4113-9d5c-84029ec53140\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") "
Oct 30 01:15:22.950495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.942785    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee09f6bd-6a43-4113-9d5c-84029ec53140-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee09f6bd-6a43-4113-9d5c-84029ec53140-maas-api-service-ca\") pod \"ee09f6bd-6a43-4113-9d5c-84029ec53140\" (UID: \"ee09f6bd-6a43-4113-9d5c-84029ec53140\") "
Oct 30 01:15:22.950495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.942853    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78645a0a-feb8-41d3-95a2-addfb511c1fd-kube-api-access-bfdz7\" (UniqueName: \"kubernetes.io/projected/78645a0a-feb8-41d3-95a2-addfb511c1fd-kube-api-access-bfdz7\") pod \"78645a0a-feb8-41d3-95a2-addfb511c1fd\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") "
Oct 30 01:15:22.950495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.942883    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78645a0a-feb8-41d3-95a2-addfb511c1fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78645a0a-feb8-41d3-95a2-addfb511c1fd-maas-api-service-ca\") pod \"78645a0a-feb8-41d3-95a2-addfb511c1fd\" (UID: \"78645a0a-feb8-41d3-95a2-addfb511c1fd\") "
Oct 30 01:15:22.950495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.946549    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78645a0a-feb8-41d3-95a2-addfb511c1fd-maas-api-service-ca" pod "78645a0a-feb8-41d3-95a2-addfb511c1fd" (UID: "78645a0a-feb8-41d3-95a2-addfb511c1fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.955567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:22.955521    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee09f6bd-6a43-4113-9d5c-84029ec53140-maas-api-service-ca" pod "ee09f6bd-6a43-4113-9d5c-84029ec53140" (UID: "ee09f6bd-6a43-4113-9d5c-84029ec53140"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:23.005688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.005183    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee09f6bd-6a43-4113-9d5c-84029ec53140-kube-api-access-h5zd9" pod "ee09f6bd-6a43-4113-9d5c-84029ec53140" (UID: "ee09f6bd-6a43-4113-9d5c-84029ec53140"). InnerVolumeSpecName "kube-api-access-h5zd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:23.006674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.006625    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78645a0a-feb8-41d3-95a2-addfb511c1fd-kube-api-access-bfdz7" pod "78645a0a-feb8-41d3-95a2-addfb511c1fd" (UID: "78645a0a-feb8-41d3-95a2-addfb511c1fd"). InnerVolumeSpecName "kube-api-access-bfdz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:23.015101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.015051    5104 scope.go:122] "RemoveContainer" containerID="018f9e289c5f236da869f077ab3c69a71089be525b57a839ff98e704dd34b05c"
Oct 30 01:15:23.037774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.037743    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:23.044242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.044171    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:23.049480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.049265    5104 scope.go:122] "RemoveContainer" containerID="94d8693b9bca24dc5c457a18aad52a00d91306c17dc8e6dc580b3cbb5f2f3a97"
Oct 30 01:15:23.053481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.053439    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee09f6bd-6a43-4113-9d5c-84029ec53140-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.053681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.053664    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdz7\" (UniqueName: \"kubernetes.io/projected/78645a0a-feb8-41d3-95a2-addfb511c1fd-kube-api-access-bfdz7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.054116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.054054    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78645a0a-feb8-41d3-95a2-addfb511c1fd-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.054116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.054096    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zd9\" (UniqueName: \"kubernetes.io/projected/ee09f6bd-6a43-4113-9d5c-84029ec53140-kube-api-access-h5zd9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.101537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.101447    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"]
Oct 30 01:15:23.107364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.107336    5104 scope.go:122] "RemoveContainer" containerID="f7ed1e5f5c1d4b7fb0c3014e4b4c1b0d555a7f63bee3a3ef9a98dda0ac887b07"
Oct 30 01:15:23.127788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.123845    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-eacea3-29888715-dsp2m"]
Oct 30 01:15:23.128509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.128183    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:23.134766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.134743    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"]
Oct 30 01:15:23.144259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.144230    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e896-29888715-dnfxg"]
Oct 30 01:15:23.155711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.155674    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0489e1db-1b58-4955-966b-82a17015a4be-kube-api-access-nb4bx\" (UniqueName: \"kubernetes.io/projected/0489e1db-1b58-4955-966b-82a17015a4be-kube-api-access-nb4bx\") pod \"0489e1db-1b58-4955-966b-82a17015a4be\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") "
Oct 30 01:15:23.155876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.155780    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0489e1db-1b58-4955-966b-82a17015a4be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0489e1db-1b58-4955-966b-82a17015a4be-maas-api-service-ca\") pod \"0489e1db-1b58-4955-966b-82a17015a4be\" (UID: \"0489e1db-1b58-4955-966b-82a17015a4be\") "
Oct 30 01:15:23.155984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.155966    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-maas-api-service-ca\") pod \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") "
Oct 30 01:15:23.156155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.156125    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-kube-api-access-tkfrk\" (UniqueName: \"kubernetes.io/projected/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-kube-api-access-tkfrk\") pod \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\" (UID: \"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16\") "
Oct 30 01:15:23.156278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.156166    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0489e1db-1b58-4955-966b-82a17015a4be-maas-api-service-ca" pod "0489e1db-1b58-4955-966b-82a17015a4be" (UID: "0489e1db-1b58-4955-966b-82a17015a4be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:23.156376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.156355    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-maas-api-service-ca" pod "1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" (UID: "1e6a9e4d-59c8-4aca-a777-c5f2510b1d16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:23.156938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.156912    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0489e1db-1b58-4955-966b-82a17015a4be-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.157041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.156940    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.158292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.158220    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-kube-api-access-tkfrk" pod "1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" (UID: "1e6a9e4d-59c8-4aca-a777-c5f2510b1d16"). InnerVolumeSpecName "kube-api-access-tkfrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:23.158617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.158590    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0489e1db-1b58-4955-966b-82a17015a4be-kube-api-access-nb4bx" pod "0489e1db-1b58-4955-966b-82a17015a4be" (UID: "0489e1db-1b58-4955-966b-82a17015a4be"). InnerVolumeSpecName "kube-api-access-nb4bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:23.163327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.163269    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"]
Oct 30 01:15:23.168692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.168669    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52f1cb-29888715-rlkjv"]
Oct 30 01:15:23.258607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.258573    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffea74c7-864a-4a9f-b52a-2edd8aea5559-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffea74c7-864a-4a9f-b52a-2edd8aea5559-maas-api-service-ca\") pod \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") "
Oct 30 01:15:23.258827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.258812    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffea74c7-864a-4a9f-b52a-2edd8aea5559-kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/ffea74c7-864a-4a9f-b52a-2edd8aea5559-kube-api-access-cpvjp\") pod \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\" (UID: \"ffea74c7-864a-4a9f-b52a-2edd8aea5559\") "
Oct 30 01:15:23.259062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.259033    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffea74c7-864a-4a9f-b52a-2edd8aea5559-maas-api-service-ca" pod "ffea74c7-864a-4a9f-b52a-2edd8aea5559" (UID: "ffea74c7-864a-4a9f-b52a-2edd8aea5559"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:23.259707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.259670    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffea74c7-864a-4a9f-b52a-2edd8aea5559-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.259816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.259712    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkfrk\" (UniqueName: \"kubernetes.io/projected/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16-kube-api-access-tkfrk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.259816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.259728    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb4bx\" (UniqueName: \"kubernetes.io/projected/0489e1db-1b58-4955-966b-82a17015a4be-kube-api-access-nb4bx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.261170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.261143    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffea74c7-864a-4a9f-b52a-2edd8aea5559-kube-api-access-cpvjp" pod "ffea74c7-864a-4a9f-b52a-2edd8aea5559" (UID: "ffea74c7-864a-4a9f-b52a-2edd8aea5559"). InnerVolumeSpecName "kube-api-access-cpvjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:23.361843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.361809    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/ffea74c7-864a-4a9f-b52a-2edd8aea5559-kube-api-access-cpvjp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.381015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.380968    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"]
Oct 30 01:15:23.389544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.389507    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-262230-29888715-znrvz"]
Oct 30 01:15:23.425235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.425203    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"]
Oct 30 01:15:23.434002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.433963    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-64f472-29888715-tlb6l"]
Oct 30 01:15:23.486795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.486757    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx" event={"ID":"ee09f6bd-6a43-4113-9d5c-84029ec53140","Type":"ContainerDied","Data":"d728f4cc898a1d6acfc0a97ecb3be07e0b0477f715697bf82a8255fed8406430"}
Oct 30 01:15:23.486978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.486775    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"
Oct 30 01:15:23.486978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.486826    5104 scope.go:122] "RemoveContainer" containerID="c4ed081ef8fb9258f44c8699820d3f562d092fe6bf1555e4f2cecf6d56c72cef"
Oct 30 01:15:23.489095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.489070    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"
Oct 30 01:15:23.489095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.489078    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c" event={"ID":"1e6a9e4d-59c8-4aca-a777-c5f2510b1d16","Type":"ContainerDied","Data":"40e253a918f313e6313a93fbc89bcef2e75a5acb0c946779f0fac5a912665884"}
Oct 30 01:15:23.492707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.492680    5104 generic.go:356] "Generic (PLEG): container finished" podID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerID="d608095ed649a54652309850d14cf8a4a5d08fcebc99190a0375ed5d535f41f6" exitCode=60
Oct 30 01:15:23.492854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.492756    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" event={"ID":"f0ae7c6c-84f1-47ab-8b4e-86a121949b97","Type":"ContainerDied","Data":"d608095ed649a54652309850d14cf8a4a5d08fcebc99190a0375ed5d535f41f6"}
Oct 30 01:15:23.493786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.493766    5104 scope.go:122] "RemoveContainer" containerID="d608095ed649a54652309850d14cf8a4a5d08fcebc99190a0375ed5d535f41f6"
Oct 30 01:15:23.494371 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:23.494313    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb_odh-ai-gateway-infra(f0ae7c6c-84f1-47ab-8b4e-86a121949b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97"
Oct 30 01:15:23.499629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.499602    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5" event={"ID":"78645a0a-feb8-41d3-95a2-addfb511c1fd","Type":"ContainerDied","Data":"2e758c0a7564f716c4b4bb50fb9528c9c1cdac9f3bb56285d4ca144f933aadcc"}
Oct 30 01:15:23.499841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.499624    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"
Oct 30 01:15:23.502724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.502623    5104 scope.go:122] "RemoveContainer" containerID="c641bde82069dda1c63cdc5441b1b90ff82a413b4a47eee793d5ceecfa030faa"
Oct 30 01:15:23.503805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.503756    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn" event={"ID":"ffea74c7-864a-4a9f-b52a-2edd8aea5559","Type":"ContainerDied","Data":"89af54181e3048c880230eeba94bca7afdafff6bb93246ab2df50e0111ce5bae"}
Oct 30 01:15:23.503895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.503799    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"
Oct 30 01:15:23.526970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.526930    5104 scope.go:122] "RemoveContainer" containerID="df11e71a259e7c050632f1af49843a4bc99841e089ca1ac959c8879ebe648fbb"
Oct 30 01:15:23.542716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.542685    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"
Oct 30 01:15:23.543546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.542690    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29" event={"ID":"0489e1db-1b58-4955-966b-82a17015a4be","Type":"ContainerDied","Data":"d6c418c5e4f5ecc6322bb007c192d91f30ff71487a3fae9ee11fa87f4eb513cf"}
Oct 30 01:15:23.550515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.550490    5104 scope.go:122] "RemoveContainer" containerID="7091cd5cded569e893258e9a0b6efcec00d69b55d507b6991440f10cacb8a971"
Oct 30 01:15:23.567920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.567894    5104 scope.go:122] "RemoveContainer" containerID="21cef5df943a8ba41007a84605a15bbb066923b0a08ce9d1839311d18bcf8e66"
Oct 30 01:15:23.584736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.584712    5104 scope.go:122] "RemoveContainer" containerID="b756b6ee6ab904abe6ebe7f93864096a2211addad011225ae4dcd880b34f1f14"
Oct 30 01:15:23.590417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.590377    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"]
Oct 30 01:15:23.672350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.672322    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"]
Oct 30 01:15:23.698930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.682069    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-56d1d9-29888715-nzdv5"]
Oct 30 01:15:23.708745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.708714    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"]
Oct 30 01:15:23.723348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.723322    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"]
Oct 30 01:15:23.728505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.728427    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-29145c-29888715-fpmzn"]
Oct 30 01:15:23.743736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.743425    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"]
Oct 30 01:15:23.757671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.757599    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"]
Oct 30 01:15:23.957530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.957439    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-18a189-29888715-7tnvx"]
Oct 30 01:15:23.976590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.976073    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"]
Oct 30 01:15:23.996055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.985830    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:24.000985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:23.999677    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:24.022338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.019828    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:24.026652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.026593    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"]
Oct 30 01:15:24.028295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.028270    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:24.036086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.034706    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d2cd1c-ddd7-419a-8507-7e355abaf111-kube-api-access-vrdqg\" (UniqueName: \"kubernetes.io/projected/c6d2cd1c-ddd7-419a-8507-7e355abaf111-kube-api-access-vrdqg\") pod \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") "
Oct 30 01:15:24.036086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.034819    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b270123-f6a5-4202-b8e3-699b84d55ee9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b270123-f6a5-4202-b8e3-699b84d55ee9-maas-api-service-ca\") pod \"8b270123-f6a5-4202-b8e3-699b84d55ee9\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") "
Oct 30 01:15:24.036086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.034862    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b270123-f6a5-4202-b8e3-699b84d55ee9-kube-api-access-ldb9k\" (UniqueName: \"kubernetes.io/projected/8b270123-f6a5-4202-b8e3-699b84d55ee9-kube-api-access-ldb9k\") pod \"8b270123-f6a5-4202-b8e3-699b84d55ee9\" (UID: \"8b270123-f6a5-4202-b8e3-699b84d55ee9\") "
Oct 30 01:15:24.036086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.034977    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d2cd1c-ddd7-419a-8507-7e355abaf111-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d2cd1c-ddd7-419a-8507-7e355abaf111-maas-api-service-ca\") pod \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\" (UID: \"c6d2cd1c-ddd7-419a-8507-7e355abaf111\") "
Oct 30 01:15:24.036086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.035438    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b270123-f6a5-4202-b8e3-699b84d55ee9-maas-api-service-ca" pod "8b270123-f6a5-4202-b8e3-699b84d55ee9" (UID: "8b270123-f6a5-4202-b8e3-699b84d55ee9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.036086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.035868    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b270123-f6a5-4202-b8e3-699b84d55ee9-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.036922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.036899    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d2cd1c-ddd7-419a-8507-7e355abaf111-maas-api-service-ca" pod "c6d2cd1c-ddd7-419a-8507-7e355abaf111" (UID: "c6d2cd1c-ddd7-419a-8507-7e355abaf111"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.037383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.037366    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888715-dx58c"]
Oct 30 01:15:24.042518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.040870    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b270123-f6a5-4202-b8e3-699b84d55ee9-kube-api-access-ldb9k" pod "8b270123-f6a5-4202-b8e3-699b84d55ee9" (UID: "8b270123-f6a5-4202-b8e3-699b84d55ee9"). InnerVolumeSpecName "kube-api-access-ldb9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.042518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.041497    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d2cd1c-ddd7-419a-8507-7e355abaf111-kube-api-access-vrdqg" pod "c6d2cd1c-ddd7-419a-8507-7e355abaf111" (UID: "c6d2cd1c-ddd7-419a-8507-7e355abaf111"). InnerVolumeSpecName "kube-api-access-vrdqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.053487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.051769    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"]
Oct 30 01:15:24.069416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.069358    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"]
Oct 30 01:15:24.080081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.080016    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-687471-29888715-4rz29"]
Oct 30 01:15:24.139158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.139129    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrdqg\" (UniqueName: \"kubernetes.io/projected/c6d2cd1c-ddd7-419a-8507-7e355abaf111-kube-api-access-vrdqg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.139713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.139649    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldb9k\" (UniqueName: \"kubernetes.io/projected/8b270123-f6a5-4202-b8e3-699b84d55ee9-kube-api-access-ldb9k\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.140280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.140262    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d2cd1c-ddd7-419a-8507-7e355abaf111-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.141788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.140899    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:24.241725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.241699    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad6b4c4-517f-41a2-b986-e7367bef493b-kube-api-access-cg4cv\" (UniqueName: \"kubernetes.io/projected/aad6b4c4-517f-41a2-b986-e7367bef493b-kube-api-access-cg4cv\") pod \"aad6b4c4-517f-41a2-b986-e7367bef493b\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") "
Oct 30 01:15:24.241980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.241961    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad6b4c4-517f-41a2-b986-e7367bef493b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad6b4c4-517f-41a2-b986-e7367bef493b-maas-api-service-ca\") pod \"aad6b4c4-517f-41a2-b986-e7367bef493b\" (UID: \"aad6b4c4-517f-41a2-b986-e7367bef493b\") "
Oct 30 01:15:24.242406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.242375    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad6b4c4-517f-41a2-b986-e7367bef493b-maas-api-service-ca" pod "aad6b4c4-517f-41a2-b986-e7367bef493b" (UID: "aad6b4c4-517f-41a2-b986-e7367bef493b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.242843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.242818    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad6b4c4-517f-41a2-b986-e7367bef493b-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.244432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.244403    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad6b4c4-517f-41a2-b986-e7367bef493b-kube-api-access-cg4cv" pod "aad6b4c4-517f-41a2-b986-e7367bef493b" (UID: "aad6b4c4-517f-41a2-b986-e7367bef493b"). InnerVolumeSpecName "kube-api-access-cg4cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.312442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.312327    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:24.344371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.344334    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a510c363-24f2-41f1-869e-a509b345d5e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510c363-24f2-41f1-869e-a509b345d5e5-maas-api-service-ca\") pod \"a510c363-24f2-41f1-869e-a509b345d5e5\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") "
Oct 30 01:15:24.345064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.345035    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a510c363-24f2-41f1-869e-a509b345d5e5-kube-api-access-67429\" (UniqueName: \"kubernetes.io/projected/a510c363-24f2-41f1-869e-a509b345d5e5-kube-api-access-67429\") pod \"a510c363-24f2-41f1-869e-a509b345d5e5\" (UID: \"a510c363-24f2-41f1-869e-a509b345d5e5\") "
Oct 30 01:15:24.345971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.345775    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a510c363-24f2-41f1-869e-a509b345d5e5-maas-api-service-ca" pod "a510c363-24f2-41f1-869e-a509b345d5e5" (UID: "a510c363-24f2-41f1-869e-a509b345d5e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.348353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.348327    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg4cv\" (UniqueName: \"kubernetes.io/projected/aad6b4c4-517f-41a2-b986-e7367bef493b-kube-api-access-cg4cv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.348523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.348501    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510c363-24f2-41f1-869e-a509b345d5e5-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.368102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.367897    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:24.373488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.370596    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:24.379618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.376635    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a510c363-24f2-41f1-869e-a509b345d5e5-kube-api-access-67429" pod "a510c363-24f2-41f1-869e-a509b345d5e5" (UID: "a510c363-24f2-41f1-869e-a509b345d5e5"). InnerVolumeSpecName "kube-api-access-67429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.449502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.449452    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-kube-api-access-v877h\" (UniqueName: \"kubernetes.io/projected/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-kube-api-access-v877h\") pod \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") "
Oct 30 01:15:24.449664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.449554    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5910f9-37eb-42cd-a09d-417340f02866-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5910f9-37eb-42cd-a09d-417340f02866-maas-api-service-ca\") pod \"8a5910f9-37eb-42cd-a09d-417340f02866\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") "
Oct 30 01:15:24.449664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.449612    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-maas-api-service-ca\") pod \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\" (UID: \"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93\") "
Oct 30 01:15:24.449781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.449735    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5910f9-37eb-42cd-a09d-417340f02866-kube-api-access-kj7nn\" (UniqueName: \"kubernetes.io/projected/8a5910f9-37eb-42cd-a09d-417340f02866-kube-api-access-kj7nn\") pod \"8a5910f9-37eb-42cd-a09d-417340f02866\" (UID: \"8a5910f9-37eb-42cd-a09d-417340f02866\") "
Oct 30 01:15:24.450000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.449969    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5910f9-37eb-42cd-a09d-417340f02866-maas-api-service-ca" pod "8a5910f9-37eb-42cd-a09d-417340f02866" (UID: "8a5910f9-37eb-42cd-a09d-417340f02866"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.450443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.450414    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-maas-api-service-ca" pod "45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" (UID: "45a664ef-0a32-4a1a-98a1-5ccfdb4eed93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.450666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.450648    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.450752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.450668    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67429\" (UniqueName: \"kubernetes.io/projected/a510c363-24f2-41f1-869e-a509b345d5e5-kube-api-access-67429\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.450752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.450679    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5910f9-37eb-42cd-a09d-417340f02866-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.452252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.452217    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-kube-api-access-v877h" pod "45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" (UID: "45a664ef-0a32-4a1a-98a1-5ccfdb4eed93"). InnerVolumeSpecName "kube-api-access-v877h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.452367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.452286    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5910f9-37eb-42cd-a09d-417340f02866-kube-api-access-kj7nn" pod "8a5910f9-37eb-42cd-a09d-417340f02866" (UID: "8a5910f9-37eb-42cd-a09d-417340f02866"). InnerVolumeSpecName "kube-api-access-kj7nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.552832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.552749    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v877h\" (UniqueName: \"kubernetes.io/projected/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93-kube-api-access-v877h\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.552832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.552781    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj7nn\" (UniqueName: \"kubernetes.io/projected/8a5910f9-37eb-42cd-a09d-417340f02866-kube-api-access-kj7nn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.561406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.561378    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"]
Oct 30 01:15:24.566625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.566602    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"
Oct 30 01:15:24.566625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.566602    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7" event={"ID":"8b270123-f6a5-4202-b8e3-699b84d55ee9","Type":"ContainerDied","Data":"4fb7220e53af08a96645c7d63f6965b47b244da636716d15ceef4836c47f197c"}
Oct 30 01:15:24.566834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.566669    5104 scope.go:122] "RemoveContainer" containerID="85fdb21c581fbd57cc18e5ecd8ae3595145c109934de23e9ee61acb53e2edd83"
Oct 30 01:15:24.578318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.577951    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"
Oct 30 01:15:24.629565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.629540    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"
Oct 30 01:15:24.631213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.631185    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"
Oct 30 01:15:24.631662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.631633    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0489e1db-1b58-4955-966b-82a17015a4be" path="/var/lib/kubelet/pods/0489e1db-1b58-4955-966b-82a17015a4be/volumes"
Oct 30 01:15:24.632445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.632420    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" path="/var/lib/kubelet/pods/1e6a9e4d-59c8-4aca-a777-c5f2510b1d16/volumes"
Oct 30 01:15:24.633566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.633092    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" path="/var/lib/kubelet/pods/25e08139-14a0-4499-aa79-6bb318c2f531/volumes"
Oct 30 01:15:24.634501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.634173    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293a0b39-9772-4823-b91c-8f99316e92db" path="/var/lib/kubelet/pods/293a0b39-9772-4823-b91c-8f99316e92db/volumes"
Oct 30 01:15:24.634855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.634831    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" path="/var/lib/kubelet/pods/4cbf3f29-d8b1-4215-8c7d-b7311a0df231/volumes"
Oct 30 01:15:24.636355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.635382    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"
Oct 30 01:15:24.636355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.635556    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" path="/var/lib/kubelet/pods/5b931cfc-6a3d-4381-b5af-035ea30563cb/volumes"
Oct 30 01:15:24.636684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.636662    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" path="/var/lib/kubelet/pods/78645a0a-feb8-41d3-95a2-addfb511c1fd/volumes"
Oct 30 01:15:24.637311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.637285    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" path="/var/lib/kubelet/pods/b1265940-ee90-4ede-9e4a-9bd690eb1681/volumes"
Oct 30 01:15:24.637943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.637920    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f66867-6e46-479b-a883-33b036fa6279" path="/var/lib/kubelet/pods/c0f66867-6e46-479b-a883-33b036fa6279/volumes"
Oct 30 01:15:24.639540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.638988    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" path="/var/lib/kubelet/pods/ee09f6bd-6a43-4113-9d5c-84029ec53140/volumes"
Oct 30 01:15:24.640208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.639628    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" path="/var/lib/kubelet/pods/ffea74c7-864a-4a9f-b52a-2edd8aea5559/volumes"
Oct 30 01:15:24.641370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.641341    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq" event={"ID":"a510c363-24f2-41f1-869e-a509b345d5e5","Type":"ContainerDied","Data":"32bd5182523d6f3f63a6cd643326c5357c71b29fea9300ef721026e2f9056a6d"}
Oct 30 01:15:24.641534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.641390    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l" event={"ID":"aad6b4c4-517f-41a2-b986-e7367bef493b","Type":"ContainerDied","Data":"863f68ca263a810000f8a87796686127d66630977ce31ddfaa6fde888a169b97"}
Oct 30 01:15:24.641534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.641409    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5" event={"ID":"45a664ef-0a32-4a1a-98a1-5ccfdb4eed93","Type":"ContainerDied","Data":"b137837464679a5f03d017e8ad5c0c5258e88aef0b62e102f8db212ca6388560"}
Oct 30 01:15:24.641908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.641446    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs" event={"ID":"c6d2cd1c-ddd7-419a-8507-7e355abaf111","Type":"ContainerDied","Data":"629ff58eb44cfc6c4fe3a13b4ccc90e8d8748367d9dc554eb82eea1cee0aba4c"}
Oct 30 01:15:24.642103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.642086    5104 scope.go:122] "RemoveContainer" containerID="4c4ed5f6ec89d10a7f3371883e4341c920033a186265a162dedb23249b8cfbf4"
Oct 30 01:15:24.645644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.645624    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"]
Oct 30 01:15:24.651579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.651527    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4dde48-29888715-7xrr7"]
Oct 30 01:15:24.662184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.662164    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"
Oct 30 01:15:24.662297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.662231    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw" event={"ID":"8a5910f9-37eb-42cd-a09d-417340f02866","Type":"ContainerDied","Data":"292346fd3376ec12ffbb44092fc188551bdce6749d2e3e61cc916836544c14b7"}
Oct 30 01:15:24.682262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.677936    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:15:24.682262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.679367    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"]
Oct 30 01:15:24.689789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.689756    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888715-l2nqq"]
Oct 30 01:15:24.738879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.738857    5104 scope.go:122] "RemoveContainer" containerID="b63d4be038f65dc8aaed0788e2bd0c5de83f32ef26929a50d2d602cc60be2a4f"
Oct 30 01:15:24.757665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.757639    5104 scope.go:122] "RemoveContainer" containerID="724c12226aaecceaa1589d4aecdc490e9ff41f87d61d52452182fcd15c003265"
Oct 30 01:15:24.772856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.772834    5104 scope.go:122] "RemoveContainer" containerID="fac52f23e4b262d1eccc5a1ea9d3f6e8e216c134e9604ca1d7cdd623704e8eb9"
Oct 30 01:15:24.788948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.788925    5104 scope.go:122] "RemoveContainer" containerID="6d03e0e714f1a7732aef6ca33939c669f7324f938127fde7ad5b3febe97c2c62"
Oct 30 01:15:24.816283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.816211    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"]
Oct 30 01:15:24.823222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.823196    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-69dc90-29888715-4hprs"]
Oct 30 01:15:24.864030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.863992    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:24.901618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.901580    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"]
Oct 30 01:15:24.912110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.912076    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50d078-29888715-nql7l"]
Oct 30 01:15:24.930260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.930227    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"]
Oct 30 01:15:24.935560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.935528    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-82e9bf-29888715-hmsz5"]
Oct 30 01:15:24.961899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.961835    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-maas-api-service-ca\") pod \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") "
Oct 30 01:15:24.962171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.962089    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-kube-api-access-4zl72\" (UniqueName: \"kubernetes.io/projected/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-kube-api-access-4zl72\") pod \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\" (UID: \"f0ae7c6c-84f1-47ab-8b4e-86a121949b97\") "
Oct 30 01:15:24.964704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.964669    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-maas-api-service-ca" pod "f0ae7c6c-84f1-47ab-8b4e-86a121949b97" (UID: "f0ae7c6c-84f1-47ab-8b4e-86a121949b97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.967656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.967623    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-kube-api-access-4zl72" pod "f0ae7c6c-84f1-47ab-8b4e-86a121949b97" (UID: "f0ae7c6c-84f1-47ab-8b4e-86a121949b97"). InnerVolumeSpecName "kube-api-access-4zl72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.973260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.973215    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"]
Oct 30 01:15:24.984212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:24.984155    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888715-l75jw"]
Oct 30 01:15:25.064865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:25.064821    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zl72\" (UniqueName: \"kubernetes.io/projected/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-kube-api-access-4zl72\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.064865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:25.064863    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ae7c6c-84f1-47ab-8b4e-86a121949b97-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.748903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:25.748872    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"
Oct 30 01:15:25.749530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:25.748972    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb" event={"ID":"f0ae7c6c-84f1-47ab-8b4e-86a121949b97","Type":"ContainerDied","Data":"2d3768466b2bf72d7f57792906d7bb597f6990aee6ddcf6168a590844b1b8994"}
Oct 30 01:15:25.750298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:25.750268    5104 scope.go:122] "RemoveContainer" containerID="d608095ed649a54652309850d14cf8a4a5d08fcebc99190a0375ed5d535f41f6"
Oct 30 01:15:25.805776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:25.805744    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"]
Oct 30 01:15:25.814252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:25.814223    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888715-xpxsb"]
Oct 30 01:15:26.585697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:26.585658    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" path="/var/lib/kubelet/pods/45a664ef-0a32-4a1a-98a1-5ccfdb4eed93/volumes"
Oct 30 01:15:26.586512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:26.586446    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" path="/var/lib/kubelet/pods/8a5910f9-37eb-42cd-a09d-417340f02866/volumes"
Oct 30 01:15:26.587181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:26.587161    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" path="/var/lib/kubelet/pods/8b270123-f6a5-4202-b8e3-699b84d55ee9/volumes"
Oct 30 01:15:26.588633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:26.588600    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" path="/var/lib/kubelet/pods/a510c363-24f2-41f1-869e-a509b345d5e5/volumes"
Oct 30 01:15:26.589274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:26.589254    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" path="/var/lib/kubelet/pods/aad6b4c4-517f-41a2-b986-e7367bef493b/volumes"
Oct 30 01:15:26.589900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:26.589879    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" path="/var/lib/kubelet/pods/c6d2cd1c-ddd7-419a-8507-7e355abaf111/volumes"
Oct 30 01:15:26.590922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:26.590902    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" path="/var/lib/kubelet/pods/f0ae7c6c-84f1-47ab-8b4e-86a121949b97/volumes"
Oct 30 01:15:35.262276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:35.262237    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:15:37.561486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:37.559625    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.232:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:37.649480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:37.648180    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:37.677484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:37.676565    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg" podUID="9a27a7b6-042c-4a01-a8a7-79afa3b7f400" containerName="main" probeResult="failure" output="Get \"https://10.132.1.5:8000/ready\": context deadline exceeded"
Oct 30 01:15:38.415488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:38.412584    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" podUID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.239:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:39.110517 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:15:39.110473    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:15:44.267419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.267373    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"]
Oct 30 01:15:44.268327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268310    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268327    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268348    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268353    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268360    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268364    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268371    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268376    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:44.268384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268385    5104 cpu_manager.go:414] "containerMap: removing container" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268389    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268395    5104 cpu_manager.go:414] "containerMap: removing container" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268399    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268406    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268411    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268420    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268424    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268430    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268434    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268440    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268444    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268466    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268470    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268490    5104 cpu_manager.go:414] "containerMap: removing container" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268495    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268504    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268508    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268514    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268518    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268524    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268529    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268537    5104 cpu_manager.go:414] "containerMap: removing container" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268541    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268550    5104 cpu_manager.go:414] "containerMap: removing container" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268554    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268562    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268566    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268573    5104 cpu_manager.go:414] "containerMap: removing container" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268577    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268585    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268590    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268595    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268600    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268610    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268614    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268620    5104 cpu_manager.go:414] "containerMap: removing container" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268624    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268630    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268635    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268643    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268648    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268652    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268656    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268665    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268670    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268677    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268681    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268687    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268692    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:44.268684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268707    5104 cpu_manager.go:414] "containerMap: removing container" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268712    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268719    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268723    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268730    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268735    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268745    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268749    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268755    5104 cpu_manager.go:414] "containerMap: removing container" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268759    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268766    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268770    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268778    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268782    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268789    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268793    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268799    5104 cpu_manager.go:414] "containerMap: removing container" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268803    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268811    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268815    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268822    5104 cpu_manager.go:414] "containerMap: removing container" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268826    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268834    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268838    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.268994    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269005    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269018    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269026    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269040    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269051    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269061    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269069    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269078    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269086    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269118    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269127    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269136    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269145    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269152    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269162    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269171    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269183    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269195    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269204    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269212    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269219    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269226    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269234    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269241    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269249    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269262    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269271    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269281    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269291    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269300    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1265940-ee90-4ede-9e4a-9bd690eb1681" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269312    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269320    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d2cd1c-ddd7-419a-8507-7e355abaf111" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269327    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269337    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269345    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269354    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269364    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269374    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269385    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269395    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269404    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:44.270818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.269415    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad6b4c4-517f-41a2-b986-e7367bef493b" containerName="cleanup"
Oct 30 01:15:44.274717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.274354    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.298982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.298864    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e5711\"/\"openshift-service-ca.crt\""
Oct 30 01:15:44.299355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.299302    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e5711\"/\"kube-root-ca.crt\""
Oct 30 01:15:44.299794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.299685    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e5711\"/\"test-model-0e5711-kserve-self-signed-certs\""
Oct 30 01:15:44.300052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.300018    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e5711\"/\"default-dockercfg-g6sz2\""
Oct 30 01:15:44.300330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.300308    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"]
Oct 30 01:15:44.391840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.391807    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tmp-dir\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.392059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.391857    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvs2\" (UniqueName: \"kubernetes.io/projected/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-kube-api-access-7nvs2\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.392059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.392037    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-model-cache\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.392195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.392162    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tls-certs\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.392247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.392217    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-dshm\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.392342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.392245    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-home\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.495172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.494872    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tmp-dir\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.495172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.494953    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvs2\" (UniqueName: \"kubernetes.io/projected/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-kube-api-access-7nvs2\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.495172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.495069    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-model-cache\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.495888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.495593    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tls-certs\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.495888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.495674    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-dshm\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.495888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.495725    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-home\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.502205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.501607    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-home\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.502205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.501886    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tmp-dir\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.503361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.503313    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-model-cache\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.529522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.528876    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tls-certs\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.537946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.537719    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-dshm\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.556851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.556814    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvs2\" (UniqueName: \"kubernetes.io/projected/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-kube-api-access-7nvs2\") pod \"test-model-0e5711-kserve-567f756789-9vkbg\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") " pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:44.608482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:44.595747    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:45.125895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:45.125858    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"]
Oct 30 01:15:45.126213 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:45.126192    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0d7b9ba_12aa_4de5_b9d2_edba23bcf767.slice/crio-054bbac424fda5019388df3522af149fed87c206fc844af10ee82db49e7fdf96 WatchSource:0}: Error finding container 054bbac424fda5019388df3522af149fed87c206fc844af10ee82db49e7fdf96: Status 404 returned error can't find the container with id 054bbac424fda5019388df3522af149fed87c206fc844af10ee82db49e7fdf96
Oct 30 01:15:45.202794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:45.202751    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg" event={"ID":"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767","Type":"ContainerStarted","Data":"054bbac424fda5019388df3522af149fed87c206fc844af10ee82db49e7fdf96"}
Oct 30 01:15:46.227004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:46.226962    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg" event={"ID":"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767","Type":"ContainerStarted","Data":"38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8"}
Oct 30 01:15:46.264952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:46.264898    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg" podStartSLOduration=2.264884411 podStartE2EDuration="2.264884411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:46.263599172 +0000 UTC m=+1987.811289667" watchObservedRunningTime="2026-10-30 01:15:46.264884411 +0000 UTC m=+1987.812574907"
Oct 30 01:15:46.329107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:46.329072    5104 scope.go:122] "RemoveContainer" containerID="3a09b8d09e6fe86c7eb0bbe04bafadd63b69923cc72f64326cb5aa45d8897327"
Oct 30 01:15:46.340845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:46.340824    5104 scope.go:122] "RemoveContainer" containerID="5f7d6a0f95f93b4eea369fc954e1afbb79221e86a2c6fd5d70e3bd364eaf642b"
Oct 30 01:15:46.352164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:46.352141    5104 scope.go:122] "RemoveContainer" containerID="35ea32023c768f2d524959e256ebcfca68f0b7535f9df352b3b775343bfd82be"
Oct 30 01:15:46.596120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:46.596093    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:50.038979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.038942    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"]
Oct 30 01:15:50.040238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040216    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:50.040238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040239    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040258    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040263    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040279    5104 cpu_manager.go:414] "containerMap: removing container" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040287    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040303    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040308    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040333    5104 cpu_manager.go:414] "containerMap: removing container" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040337    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040348    5104 cpu_manager.go:414] "containerMap: removing container" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040352    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293a0b39-9772-4823-b91c-8f99316e92db" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040365    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040372    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a510c363-24f2-41f1-869e-a509b345d5e5" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040385    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040390    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040400    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:50.040401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040408    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b270123-f6a5-4202-b8e3-699b84d55ee9" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040415    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040420    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040429    5104 cpu_manager.go:414] "containerMap: removing container" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040433    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e08139-14a0-4499-aa79-6bb318c2f531" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040443    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040451    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040472    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040477    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040650    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffea74c7-864a-4a9f-b52a-2edd8aea5559" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040661    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b931cfc-6a3d-4381-b5af-035ea30563cb" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040667    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee09f6bd-6a43-4113-9d5c-84029ec53140" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040681    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbf3f29-d8b1-4215-8c7d-b7311a0df231" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040693    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6a9e4d-59c8-4aca-a777-c5f2510b1d16" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040705    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0489e1db-1b58-4955-966b-82a17015a4be" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040713    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f66867-6e46-479b-a883-33b036fa6279" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040721    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ae7c6c-84f1-47ab-8b4e-86a121949b97" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040730    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="78645a0a-feb8-41d3-95a2-addfb511c1fd" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040739    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a664ef-0a32-4a1a-98a1-5ccfdb4eed93" containerName="cleanup"
Oct 30 01:15:50.041023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.040748    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5910f9-37eb-42cd-a09d-417340f02866" containerName="cleanup"
Oct 30 01:15:50.046826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.046803    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.074037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.074008    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82e9bf-openshift-default-dockercfg-lnnf7\""
Oct 30 01:15:50.082553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.082512    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"]
Oct 30 01:15:50.176100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.176067    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-credential-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.176439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.176124    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.176810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.176784    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-podinfo\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.176980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.176952    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-certs\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.177095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.177079    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-envoy\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.177163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.177137    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istiod-ca-cert\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.177281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.177266    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-data\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.177343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.177330    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-token\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.177444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.177427    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2qf\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-kube-api-access-fr2qf\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.279606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.279566    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-envoy\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.279831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.279637    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istiod-ca-cert\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.279831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.279747    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-data\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.279831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.279797    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-token\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.279877    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2qf\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-kube-api-access-fr2qf\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.279974    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-credential-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280029    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280178    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-podinfo\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280234    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-certs\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280608    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istiod-ca-cert\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280622    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-data\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280641    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-certs\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280739    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.280926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.280903    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-credential-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.282775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.282742    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-envoy\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.283046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.283021    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-podinfo\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.299958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.299879    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-token\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.302003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.301974    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2qf\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-kube-api-access-fr2qf\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.368096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.368060    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:50.697266 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:50.697222    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952fc7a6_e26d_4361_ae4a_d2ce13284a30.slice/crio-56a0e46dbd678ca7698e0e2838a5291b20c9e1b1761c9b19241fbcefbbd8eaa0 WatchSource:0}: Error finding container 56a0e46dbd678ca7698e0e2838a5291b20c9e1b1761c9b19241fbcefbbd8eaa0: Status 404 returned error can't find the container with id 56a0e46dbd678ca7698e0e2838a5291b20c9e1b1761c9b19241fbcefbbd8eaa0
Oct 30 01:15:50.698437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.698396    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"]
Oct 30 01:15:50.701143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.701102    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:50.701276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.701199    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:50.701276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:50.701250    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:51.276963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:51.276924    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" event={"ID":"952fc7a6-e26d-4361-ae4a-d2ce13284a30","Type":"ContainerStarted","Data":"2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438"}
Oct 30 01:15:51.277389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:51.276990    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" event={"ID":"952fc7a6-e26d-4361-ae4a-d2ce13284a30","Type":"ContainerStarted","Data":"56a0e46dbd678ca7698e0e2838a5291b20c9e1b1761c9b19241fbcefbbd8eaa0"}
Oct 30 01:15:51.360237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:51.360106    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podStartSLOduration=1.360083396 podStartE2EDuration="1.360083396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:51.343189415 +0000 UTC m=+1992.890879936" watchObservedRunningTime="2026-10-30 01:15:51.360083396 +0000 UTC m=+1992.907773892"
Oct 30 01:15:51.379523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:51.379483    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:51.380039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:51.380010    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:15:51.380274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:51.380250    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:15:52.368943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:52.368906    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:15:52.369355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:52.369006    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:15:53.368431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:53.368387    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:15:53.368671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:53.368474    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:15:54.368809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:54.368770    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:15:54.369218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:54.368848    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:15:54.649532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:54.645049    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:54.676483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:54.668865    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:55.366470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.366424    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:15:55.368935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.368905    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:15:55.369389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.368972    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:15:55.551396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.551348    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"]
Oct 30 01:15:55.564046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.564016    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"]
Oct 30 01:15:55.564349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.564333    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.596839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.596801    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50d078-openshift-default-dockercfg-v9gcb\""
Oct 30 01:15:55.656833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.656438    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-token\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.656833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.656557    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-credential-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.656833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.656588    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-envoy\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.656833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.656636    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-certs\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.656833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.656723    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsnp8\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-kube-api-access-fsnp8\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.657449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.657411    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-podinfo\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.657850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.657831    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-data\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.657951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.657913    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.657951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.657931    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istiod-ca-cert\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.665255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.665225    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"]
Oct 30 01:15:55.665743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.665691    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy" containerID="cri-o://8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f" gracePeriod=30
Oct 30 01:15:55.694027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.693988    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"]
Oct 30 01:15:55.694614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.694582    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy" containerID="cri-o://df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a" gracePeriod=30
Oct 30 01:15:55.760316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.760255    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-data\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.761278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.761236    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.761530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.761503    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istiod-ca-cert\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.761837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.761805    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-token\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.764911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.764862    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.765629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.765595    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istiod-ca-cert\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.770440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.770400    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-data\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.770690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.770669    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-credential-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.771510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.771109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-credential-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.771886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.771831    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-envoy\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.780633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.780604    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-certs\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.789483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.780930    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsnp8\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-kube-api-access-fsnp8\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.790995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.790967    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-podinfo\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.828632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.782045    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-certs\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.842600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.840558    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-podinfo\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.842792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.842713    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsnp8\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-kube-api-access-fsnp8\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.851943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.851905    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-envoy\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.855274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.855242    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-token\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:55.919144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:55.919097    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:56.307850 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:15:56.307795    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7086e18_9dcd_48b3_9f26_24aa1b24cb1f.slice/crio-71e4c609169fc113958a04bbce66997c2f7637bb0ffc42ea85507411182a7a19 WatchSource:0}: Error finding container 71e4c609169fc113958a04bbce66997c2f7637bb0ffc42ea85507411182a7a19: Status 404 returned error can't find the container with id 71e4c609169fc113958a04bbce66997c2f7637bb0ffc42ea85507411182a7a19
Oct 30 01:15:56.310292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:56.310229    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"]
Oct 30 01:15:56.311738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:56.311689    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:56.311939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:56.311924    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:56.312036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:56.312024    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:15:56.368320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:56.368282    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" event={"ID":"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f","Type":"ContainerStarted","Data":"71e4c609169fc113958a04bbce66997c2f7637bb0ffc42ea85507411182a7a19"}
Oct 30 01:15:56.368538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:56.368403    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:15:56.368538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:56.368445    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:15:57.369017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:57.368975    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 01:15:57.369686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:57.369637    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 01:15:57.412848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:57.412807    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" event={"ID":"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f","Type":"ContainerStarted","Data":"4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c"}
Oct 30 01:15:57.468482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:57.466286    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podStartSLOduration=2.466265543 podStartE2EDuration="2.466265543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:57.463997103 +0000 UTC m=+1999.011687600" watchObservedRunningTime="2026-10-30 01:15:57.466265543 +0000 UTC m=+1999.013956055"
Oct 30 01:15:57.920036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:57.919998    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:15:57.921681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:57.921652    5104 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 01:15:57.921812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:57.921705    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 01:15:58.384537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:58.383794    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:58.388479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:58.385599    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:58.393482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:58.389666    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:15:58.919637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:58.919598    5104 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 01:15:58.919825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:58.919665    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 01:15:59.920482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:59.920421    5104 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 01:15:59.920871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:15:59.920512    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 01:16:00.920360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:00.920323    5104 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 01:16:00.920660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:00.920632    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 01:16:01.154183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.154151    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 01:16:01.268228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.267893    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 01:16:01.348030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.347939    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/103e840b-ae82-497a-9c10-e123a3e57e32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e840b-ae82-497a-9c10-e123a3e57e32-istio-podinfo\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.348289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.348267    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-credential-socket\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.348486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.348450    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-socket\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.348656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.348636    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-podinfo\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.348783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.348767    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-envoy\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.348921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.348905    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-socket\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.349079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.349060    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103e840b-ae82-497a-9c10-e123a3e57e32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e840b-ae82-497a-9c10-e123a3e57e32-istiod-ca-cert\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.349211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.349196    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-certs\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.349350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.349333    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-token\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-token\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.349492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.349475    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-data\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.350172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350150    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50aeb239-a767-4b4c-9263-e28d77d951bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aeb239-a767-4b4c-9263-e28d77d951bd-istiod-ca-cert\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.350387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350350    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-kube-api-access-bbf6k\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-kube-api-access-bbf6k\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.350579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350562    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-certs\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.350674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350604    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-envoy\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.350674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350635    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-istio-token\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-istio-token\") pod \"103e840b-ae82-497a-9c10-e123a3e57e32\" (UID: \"103e840b-ae82-497a-9c10-e123a3e57e32\") "
Oct 30 01:16:01.350791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350719    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-kube-api-access-hpfvw\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-kube-api-access-hpfvw\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.350850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350791    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-data\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.350908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350853    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-credential-socket\") pod \"50aeb239-a767-4b4c-9263-e28d77d951bd\" (UID: \"50aeb239-a767-4b4c-9263-e28d77d951bd\") "
Oct 30 01:16:01.351534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.350165    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-credential-socket" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.351839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.351797    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.352004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.351981    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-socket" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.352438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.352416    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103e840b-ae82-497a-9c10-e123a3e57e32-istiod-ca-cert" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:01.352863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.352842    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-certs" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.354405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.354143    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-certs" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.354765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.354742    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-socket" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.355150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.355131    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-data" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.358819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.357408    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-credential-socket" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.358819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.357637    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-data" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.358819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.357939    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50aeb239-a767-4b4c-9263-e28d77d951bd-istiod-ca-cert" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:01.359935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.359906    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-podinfo" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:16:01.360117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.360079    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-istio-token" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:01.360117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.360087    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-envoy" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.360524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.360498    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-token" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:01.360664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.360644    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-kube-api-access-bbf6k" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "kube-api-access-bbf6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:01.363170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.363146    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-kube-api-access-hpfvw" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "kube-api-access-hpfvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:01.410996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.409289    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-envoy" pod "50aeb239-a767-4b4c-9263-e28d77d951bd" (UID: "50aeb239-a767-4b4c-9263-e28d77d951bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:01.411645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.411595    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103e840b-ae82-497a-9c10-e123a3e57e32-istio-podinfo" pod "103e840b-ae82-497a-9c10-e123a3e57e32" (UID: "103e840b-ae82-497a-9c10-e123a3e57e32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462076    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462113    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462128    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462154    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfvw\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-kube-api-access-hpfvw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462172    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462186    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462199    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e840b-ae82-497a-9c10-e123a3e57e32-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462213    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462236    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462247    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462259    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462273    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e840b-ae82-497a-9c10-e123a3e57e32-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462287    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aeb239-a767-4b4c-9263-e28d77d951bd-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462300    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aeb239-a767-4b4c-9263-e28d77d951bd-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462312    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e840b-ae82-497a-9c10-e123a3e57e32-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462326    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aeb239-a767-4b4c-9263-e28d77d951bd-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.463710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.462339    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbf6k\" (UniqueName: \"kubernetes.io/projected/103e840b-ae82-497a-9c10-e123a3e57e32-kube-api-access-bbf6k\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:01.501635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.501535    5104 generic.go:356] "Generic (PLEG): container finished" podID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerID="df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a" exitCode=0
Oct 30 01:16:01.502106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.502058    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" event={"ID":"50aeb239-a767-4b4c-9263-e28d77d951bd","Type":"ContainerDied","Data":"df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a"}
Oct 30 01:16:01.502415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.502314    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w" event={"ID":"50aeb239-a767-4b4c-9263-e28d77d951bd","Type":"ContainerDied","Data":"f8fd9fc880a40d8422771d946abab7a092485acabce83942a2ac78de469e44aa"}
Oct 30 01:16:01.502415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.502382    5104 scope.go:122] "RemoveContainer" containerID="df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a"
Oct 30 01:16:01.503068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.502815    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"
Oct 30 01:16:01.554522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.553271    5104 generic.go:356] "Generic (PLEG): container finished" podID="103e840b-ae82-497a-9c10-e123a3e57e32" containerID="8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f" exitCode=0
Oct 30 01:16:01.554522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.553351    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" event={"ID":"103e840b-ae82-497a-9c10-e123a3e57e32","Type":"ContainerDied","Data":"8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f"}
Oct 30 01:16:01.554522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.553408    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k" event={"ID":"103e840b-ae82-497a-9c10-e123a3e57e32","Type":"ContainerDied","Data":"e1e316187634f732af6359a99f49c13a330cc26fb25ada8b09bb40c9b831d3b2"}
Oct 30 01:16:01.554522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.553521    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"
Oct 30 01:16:01.663381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.663349    5104 scope.go:122] "RemoveContainer" containerID="df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a"
Oct 30 01:16:01.663983 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:16:01.663819    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a\": container with ID starting with df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a not found: ID does not exist" containerID="df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a"
Oct 30 01:16:01.663983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.663859    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a"} err="failed to get container status \"df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a\": rpc error: code = NotFound desc = could not find container \"df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a\": container with ID starting with df4326242c333a793cbbcb41e95a56090e8129d10bb2cc060f17bcff5789334a not found: ID does not exist"
Oct 30 01:16:01.663983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.663886    5104 scope.go:122] "RemoveContainer" containerID="8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f"
Oct 30 01:16:01.711196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.710966    5104 scope.go:122] "RemoveContainer" containerID="8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f"
Oct 30 01:16:01.716682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.715998    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"]
Oct 30 01:16:01.719157 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:16:01.719039    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f\": container with ID starting with 8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f not found: ID does not exist" containerID="8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f"
Oct 30 01:16:01.719157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.719096    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f"} err="failed to get container status \"8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f\": rpc error: code = NotFound desc = could not find container \"8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f\": container with ID starting with 8aed3791809a71b595700240fde04d269781023a091eee60e3541d027540958f not found: ID does not exist"
Oct 30 01:16:01.743856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.743827    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbh6w"]
Oct 30 01:16:01.749648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.749610    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"]
Oct 30 01:16:01.754259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.754237    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bq26k"]
Oct 30 01:16:01.919843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.919810    5104 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 01:16:01.920028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:01.919871    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 01:16:02.410614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.410517    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"]
Oct 30 01:16:02.412955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.412919    5104 cpu_manager.go:414] "containerMap: removing container" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy"
Oct 30 01:16:02.412955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.412953    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy"
Oct 30 01:16:02.413209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.413048    5104 cpu_manager.go:414] "containerMap: removing container" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy"
Oct 30 01:16:02.413209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.413058    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy"
Oct 30 01:16:02.413358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.413338    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" containerName="istio-proxy"
Oct 30 01:16:02.413422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.413375    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" containerName="istio-proxy"
Oct 30 01:16:02.419958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.419811    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"]
Oct 30 01:16:02.419958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.419952    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.434599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.434545    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:02.435034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.434885    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:02.435034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.434980    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:16:02.446065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.446032    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:02.521487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.521427    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4zdx\" (UniqueName: \"kubernetes.io/projected/6c4b7315-a1ec-4bab-bce7-96395353bd5d-kube-api-access-p4zdx\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.522112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.522081    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.522534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.522508    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.614625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.614588    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103e840b-ae82-497a-9c10-e123a3e57e32" path="/var/lib/kubelet/pods/103e840b-ae82-497a-9c10-e123a3e57e32/volumes"
Oct 30 01:16:02.615407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.615389    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50aeb239-a767-4b4c-9263-e28d77d951bd" path="/var/lib/kubelet/pods/50aeb239-a767-4b4c-9263-e28d77d951bd/volumes"
Oct 30 01:16:02.623821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.623786    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.624203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.624171    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.624890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.624860    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4zdx\" (UniqueName: \"kubernetes.io/projected/6c4b7315-a1ec-4bab-bce7-96395353bd5d-kube-api-access-p4zdx\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.625112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.625090    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.626965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.626943    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.643720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.643681    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4zdx\" (UniqueName: \"kubernetes.io/projected/6c4b7315-a1ec-4bab-bce7-96395353bd5d-kube-api-access-p4zdx\") pod \"maas-api-revoke-sub-6567391a-hjzbd\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.741692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.741641    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:02.919890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.919851    5104 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 01:16:02.920185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:02.919910    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 01:16:03.058278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:03.058230    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"]
Oct 30 01:16:03.058924 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:03.058884    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4b7315_a1ec_4bab_bce7_96395353bd5d.slice/crio-cfe1855bfe271b67ada8f4af86f8f97195b709065ffe152da62ff1b2de0c7401 WatchSource:0}: Error finding container cfe1855bfe271b67ada8f4af86f8f97195b709065ffe152da62ff1b2de0c7401: Status 404 returned error can't find the container with id cfe1855bfe271b67ada8f4af86f8f97195b709065ffe152da62ff1b2de0c7401
Oct 30 01:16:03.578631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:03.578575    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd" event={"ID":"6c4b7315-a1ec-4bab-bce7-96395353bd5d","Type":"ContainerStarted","Data":"cfe1855bfe271b67ada8f4af86f8f97195b709065ffe152da62ff1b2de0c7401"}
Oct 30 01:16:03.926178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:03.926145    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:16:03.926856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:03.926667    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:16:03.927544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:03.927514    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:16:04.590991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:04.590900    5104 generic.go:356] "Generic (PLEG): container finished" podID="6c4b7315-a1ec-4bab-bce7-96395353bd5d" containerID="65108e5c1cf269bc8dd3dc5110799b6ca27cb9c353da97dc6cd5e3ed59a64e19" exitCode=0
Oct 30 01:16:04.590991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:04.590978    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd" event={"ID":"6c4b7315-a1ec-4bab-bce7-96395353bd5d","Type":"ContainerDied","Data":"65108e5c1cf269bc8dd3dc5110799b6ca27cb9c353da97dc6cd5e3ed59a64e19"}
Oct 30 01:16:05.838672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.838644    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:05.906032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.905954    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4b7315-a1ec-4bab-bce7-96395353bd5d-kube-api-access-p4zdx\" (UniqueName: \"kubernetes.io/projected/6c4b7315-a1ec-4bab-bce7-96395353bd5d-kube-api-access-p4zdx\") pod \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") "
Oct 30 01:16:05.906032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.906004    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-openshift-ca\") pod \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") "
Oct 30 01:16:05.906284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.906082    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-xks-ca\") pod \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\" (UID: \"6c4b7315-a1ec-4bab-bce7-96395353bd5d\") "
Oct 30 01:16:05.906695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.906662    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-openshift-ca" pod "6c4b7315-a1ec-4bab-bce7-96395353bd5d" (UID: "6c4b7315-a1ec-4bab-bce7-96395353bd5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:05.907055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.907028    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:05.908954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.908775    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-xks-ca" pod "6c4b7315-a1ec-4bab-bce7-96395353bd5d" (UID: "6c4b7315-a1ec-4bab-bce7-96395353bd5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:05.908954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:05.908914    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4b7315-a1ec-4bab-bce7-96395353bd5d-kube-api-access-p4zdx" pod "6c4b7315-a1ec-4bab-bce7-96395353bd5d" (UID: "6c4b7315-a1ec-4bab-bce7-96395353bd5d"). InnerVolumeSpecName "kube-api-access-p4zdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:06.008799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.008770    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4zdx\" (UniqueName: \"kubernetes.io/projected/6c4b7315-a1ec-4bab-bce7-96395353bd5d-kube-api-access-p4zdx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:06.008799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.008799    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4b7315-a1ec-4bab-bce7-96395353bd5d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:06.452249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.452202    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"]
Oct 30 01:16:06.454058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.454030    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6c4b7315-a1ec-4bab-bce7-96395353bd5d" containerName="revoke-keys"
Oct 30 01:16:06.454058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.454057    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4b7315-a1ec-4bab-bce7-96395353bd5d" containerName="revoke-keys"
Oct 30 01:16:06.454326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.454314    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4b7315-a1ec-4bab-bce7-96395353bd5d" containerName="revoke-keys"
Oct 30 01:16:06.459176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.459154    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.473145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.473120    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e69dee-openshift-default-dockercfg-lxjwb\""
Oct 30 01:16:06.490598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.490575    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"]
Oct 30 01:16:06.517593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.517563    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.517753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.517692    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9798f862-937c-4699-9a5b-f35362f1458e-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.517859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.517757    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.517930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.517890    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.517998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.517935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsp8\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-kube-api-access-vjsp8\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.517998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.517975    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9798f862-937c-4699-9a5b-f35362f1458e-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.518173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.518153    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.518266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.518249    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.518363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.518341    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.614076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.614047    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"
Oct 30 01:16:06.614231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.614072    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd" event={"ID":"6c4b7315-a1ec-4bab-bce7-96395353bd5d","Type":"ContainerDied","Data":"cfe1855bfe271b67ada8f4af86f8f97195b709065ffe152da62ff1b2de0c7401"}
Oct 30 01:16:06.614231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.614126    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfe1855bfe271b67ada8f4af86f8f97195b709065ffe152da62ff1b2de0c7401"
Oct 30 01:16:06.619411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619386    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.619411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619423    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsp8\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-kube-api-access-vjsp8\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.619627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619449    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9798f862-937c-4699-9a5b-f35362f1458e-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.619627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619584    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.619753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619668    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.619813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619777    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.619927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619827    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.619927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619898    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.620039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.619960    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9798f862-937c-4699-9a5b-f35362f1458e-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.620039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.620007    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.620039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.620016    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.620372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.620350    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.620417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.620378    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.620417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.620355    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9798f862-937c-4699-9a5b-f35362f1458e-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.622646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.622616    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.622936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.622915    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9798f862-937c-4699-9a5b-f35362f1458e-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.639048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.639006    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsp8\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-kube-api-access-vjsp8\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.639156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.639109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:06.787531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:06.785387    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:07.159533 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:07.159496    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9798f862_937c_4699_9a5b_f35362f1458e.slice/crio-143db26276dde22c045c506871d50107838aa0ff55a7ac9e6a7f721805623e77 WatchSource:0}: Error finding container 143db26276dde22c045c506871d50107838aa0ff55a7ac9e6a7f721805623e77: Status 404 returned error can't find the container with id 143db26276dde22c045c506871d50107838aa0ff55a7ac9e6a7f721805623e77
Oct 30 01:16:07.160911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.160887    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"]
Oct 30 01:16:07.163311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.163279    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:16:07.163428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.163347    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:16:07.163428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.163377    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:16:07.626003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.625967    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" event={"ID":"9798f862-937c-4699-9a5b-f35362f1458e","Type":"ContainerStarted","Data":"0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d"}
Oct 30 01:16:07.626179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.626027    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" event={"ID":"9798f862-937c-4699-9a5b-f35362f1458e","Type":"ContainerStarted","Data":"143db26276dde22c045c506871d50107838aa0ff55a7ac9e6a7f721805623e77"}
Oct 30 01:16:07.669870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.669818    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podStartSLOduration=1.669804431 podStartE2EDuration="1.669804431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:07.66472225 +0000 UTC m=+2009.212412747" watchObservedRunningTime="2026-10-30 01:16:07.669804431 +0000 UTC m=+2009.217494927"
Oct 30 01:16:07.790489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:07.787557    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:08.798504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:08.797645    5104 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 01:16:08.798504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:08.797728    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 01:16:09.135313 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:16:09.135233    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:16:09.793482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:09.789939    5104 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 01:16:09.793482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:09.790009    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 01:16:10.791652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:10.791610    5104 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 01:16:10.792198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:10.791677    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 01:16:11.794481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:11.788500    5104 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 01:16:11.794481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:11.788580    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 01:16:12.796481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:12.795846    5104 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 01:16:12.796481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:12.795912    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 01:16:13.786123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.786082    5104 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 01:16:13.786362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.786164    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 01:16:13.791980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.791948    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"]
Oct 30 01:16:13.797298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.797270    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.815826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.815791    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"]
Oct 30 01:16:13.815984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.815894    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:13.819328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.819275    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:13.819328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.819275    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:16:13.819600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.819347    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:13.872753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.872714    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpr6\" (UniqueName: \"kubernetes.io/projected/63901c75-9057-45af-af45-92c74df0cbfb-kube-api-access-vjpr6\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.872996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.872966    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.873171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.873151    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.975527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.975485    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpr6\" (UniqueName: \"kubernetes.io/projected/63901c75-9057-45af-af45-92c74df0cbfb-kube-api-access-vjpr6\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.975736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.975643    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.975827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.975782    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.976850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.976815    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.978378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.978345    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:13.994860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:13.994794    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpr6\" (UniqueName: \"kubernetes.io/projected/63901c75-9057-45af-af45-92c74df0cbfb-kube-api-access-vjpr6\") pod \"maas-api-revoke-sub-f33753a2-kbqts\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:14.029915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.029873    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"]
Oct 30 01:16:14.034751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.034720    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83ee24d-cvmf9"]
Oct 30 01:16:14.115296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.115206    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:14.435641 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:14.435606    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63901c75_9057_45af_af45_92c74df0cbfb.slice/crio-950572c4e0f70aa9c8caeee3500f926e008964c7e1875586cf4ea3155caa209a WatchSource:0}: Error finding container 950572c4e0f70aa9c8caeee3500f926e008964c7e1875586cf4ea3155caa209a: Status 404 returned error can't find the container with id 950572c4e0f70aa9c8caeee3500f926e008964c7e1875586cf4ea3155caa209a
Oct 30 01:16:14.436322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.436291    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"]
Oct 30 01:16:14.585521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.585481    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efc5707-f19e-48d8-94e4-c2c8962685fc" path="/var/lib/kubelet/pods/1efc5707-f19e-48d8-94e4-c2c8962685fc/volumes"
Oct 30 01:16:14.696918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.696878    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts" event={"ID":"63901c75-9057-45af-af45-92c74df0cbfb","Type":"ContainerStarted","Data":"950572c4e0f70aa9c8caeee3500f926e008964c7e1875586cf4ea3155caa209a"}
Oct 30 01:16:14.791833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.791799    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:14.792314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.792291    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:14.793204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:14.793176    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:16:15.723190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:15.723146    5104 generic.go:356] "Generic (PLEG): container finished" podID="63901c75-9057-45af-af45-92c74df0cbfb" containerID="7f9987a9a9a933a5956418aadcff2d881e5abf07b66ffcb2852c86d11422b771" exitCode=0
Oct 30 01:16:15.723777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:15.723229    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts" event={"ID":"63901c75-9057-45af-af45-92c74df0cbfb","Type":"ContainerDied","Data":"7f9987a9a9a933a5956418aadcff2d881e5abf07b66ffcb2852c86d11422b771"}
Oct 30 01:16:17.080123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.080091    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:17.168048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.168011    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-openshift-ca\") pod \"63901c75-9057-45af-af45-92c74df0cbfb\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") "
Oct 30 01:16:17.168263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.168166    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-xks-ca\") pod \"63901c75-9057-45af-af45-92c74df0cbfb\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") "
Oct 30 01:16:17.168263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.168233    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63901c75-9057-45af-af45-92c74df0cbfb-kube-api-access-vjpr6\" (UniqueName: \"kubernetes.io/projected/63901c75-9057-45af-af45-92c74df0cbfb-kube-api-access-vjpr6\") pod \"63901c75-9057-45af-af45-92c74df0cbfb\" (UID: \"63901c75-9057-45af-af45-92c74df0cbfb\") "
Oct 30 01:16:17.168416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.168385    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-openshift-ca" pod "63901c75-9057-45af-af45-92c74df0cbfb" (UID: "63901c75-9057-45af-af45-92c74df0cbfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:17.169243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.169216    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:17.171003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.170976    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-xks-ca" pod "63901c75-9057-45af-af45-92c74df0cbfb" (UID: "63901c75-9057-45af-af45-92c74df0cbfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:17.171106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.171024    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63901c75-9057-45af-af45-92c74df0cbfb-kube-api-access-vjpr6" pod "63901c75-9057-45af-af45-92c74df0cbfb" (UID: "63901c75-9057-45af-af45-92c74df0cbfb"). InnerVolumeSpecName "kube-api-access-vjpr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:17.270987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.270947    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63901c75-9057-45af-af45-92c74df0cbfb-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:17.270987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.270983    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjpr6\" (UniqueName: \"kubernetes.io/projected/63901c75-9057-45af-af45-92c74df0cbfb-kube-api-access-vjpr6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:17.742392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.742352    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"
Oct 30 01:16:17.742392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.742374    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts" event={"ID":"63901c75-9057-45af-af45-92c74df0cbfb","Type":"ContainerDied","Data":"950572c4e0f70aa9c8caeee3500f926e008964c7e1875586cf4ea3155caa209a"}
Oct 30 01:16:17.742675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:17.742425    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="950572c4e0f70aa9c8caeee3500f926e008964c7e1875586cf4ea3155caa209a"
Oct 30 01:16:26.030526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:26.030489    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"]
Oct 30 01:16:26.041379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:26.041335    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c0ed29-zcjrg"]
Oct 30 01:16:26.590290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:26.590250    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db452e21-afc3-4265-8ac9-6decf011fbdc" path="/var/lib/kubelet/pods/db452e21-afc3-4265-8ac9-6decf011fbdc/volumes"
Oct 30 01:16:31.424607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.424570    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"]
Oct 30 01:16:31.425599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.425581    5104 cpu_manager.go:414] "containerMap: removing container" podUID="63901c75-9057-45af-af45-92c74df0cbfb" containerName="revoke-keys"
Oct 30 01:16:31.425651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.425600    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63901c75-9057-45af-af45-92c74df0cbfb" containerName="revoke-keys"
Oct 30 01:16:31.425767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.425756    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="63901c75-9057-45af-af45-92c74df0cbfb" containerName="revoke-keys"
Oct 30 01:16:31.429841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.429822    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.447584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.447556    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"]
Oct 30 01:16:31.452994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.452964    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:31.453163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.453031    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:31.453163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.453056    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:31.455120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.455086    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:16:31.634402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.634360    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jc7t\" (UniqueName: \"kubernetes.io/projected/07a35d71-6e8d-4846-b7d1-c8008ebe9657-kube-api-access-6jc7t\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.634658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.634636    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.634714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.634699    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.736579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.736483    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.736579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.736569    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.736845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.736705    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jc7t\" (UniqueName: \"kubernetes.io/projected/07a35d71-6e8d-4846-b7d1-c8008ebe9657-kube-api-access-6jc7t\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.737491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.737447    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.738993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.738972    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:31.755120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:31.755092    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jc7t\" (UniqueName: \"kubernetes.io/projected/07a35d71-6e8d-4846-b7d1-c8008ebe9657-kube-api-access-6jc7t\") pod \"maas-api-revoke-sub-28c294ca-hjjq7\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:32.045924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:32.045842    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:32.359734 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:32.359689    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a35d71_6e8d_4846_b7d1_c8008ebe9657.slice/crio-1fcb0b9663306dc69e10e0bf5d5e41e24ecd657f62660d2a572ccbef8c8e6275 WatchSource:0}: Error finding container 1fcb0b9663306dc69e10e0bf5d5e41e24ecd657f62660d2a572ccbef8c8e6275: Status 404 returned error can't find the container with id 1fcb0b9663306dc69e10e0bf5d5e41e24ecd657f62660d2a572ccbef8c8e6275
Oct 30 01:16:32.360613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:32.360586    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"]
Oct 30 01:16:33.041582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:33.041488    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7" event={"ID":"07a35d71-6e8d-4846-b7d1-c8008ebe9657","Type":"ContainerStarted","Data":"26d7d84623e5fab9b1f420f392ea770d49ab5b02767ec18b8bbfd0918a1face2"}
Oct 30 01:16:33.041582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:33.041561    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7" event={"ID":"07a35d71-6e8d-4846-b7d1-c8008ebe9657","Type":"ContainerStarted","Data":"1fcb0b9663306dc69e10e0bf5d5e41e24ecd657f62660d2a572ccbef8c8e6275"}
Oct 30 01:16:33.081809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:33.081756    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7" podStartSLOduration=2.081741013 podStartE2EDuration="2.081741013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:33.080822898 +0000 UTC m=+2034.628513396" watchObservedRunningTime="2026-10-30 01:16:33.081741013 +0000 UTC m=+2034.629431508"
Oct 30 01:16:34.051942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:34.051901    5104 generic.go:356] "Generic (PLEG): container finished" podID="07a35d71-6e8d-4846-b7d1-c8008ebe9657" containerID="26d7d84623e5fab9b1f420f392ea770d49ab5b02767ec18b8bbfd0918a1face2" exitCode=0
Oct 30 01:16:34.052534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:34.051981    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7" event={"ID":"07a35d71-6e8d-4846-b7d1-c8008ebe9657","Type":"ContainerDied","Data":"26d7d84623e5fab9b1f420f392ea770d49ab5b02767ec18b8bbfd0918a1face2"}
Oct 30 01:16:35.280960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.280933    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:35.411320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.411230    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a35d71-6e8d-4846-b7d1-c8008ebe9657-kube-api-access-6jc7t\" (UniqueName: \"kubernetes.io/projected/07a35d71-6e8d-4846-b7d1-c8008ebe9657-kube-api-access-6jc7t\") pod \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") "
Oct 30 01:16:35.411564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.411426    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-xks-ca\") pod \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") "
Oct 30 01:16:35.411564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.411559    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-openshift-ca\") pod \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\" (UID: \"07a35d71-6e8d-4846-b7d1-c8008ebe9657\") "
Oct 30 01:16:35.411918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.411885    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-openshift-ca" pod "07a35d71-6e8d-4846-b7d1-c8008ebe9657" (UID: "07a35d71-6e8d-4846-b7d1-c8008ebe9657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:35.412307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.412284    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:35.413829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.413802    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-xks-ca" pod "07a35d71-6e8d-4846-b7d1-c8008ebe9657" (UID: "07a35d71-6e8d-4846-b7d1-c8008ebe9657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:35.413982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.413960    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a35d71-6e8d-4846-b7d1-c8008ebe9657-kube-api-access-6jc7t" pod "07a35d71-6e8d-4846-b7d1-c8008ebe9657" (UID: "07a35d71-6e8d-4846-b7d1-c8008ebe9657"). InnerVolumeSpecName "kube-api-access-6jc7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:35.512785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.512739    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"]
Oct 30 01:16:35.513824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.513800    5104 cpu_manager.go:414] "containerMap: removing container" podUID="07a35d71-6e8d-4846-b7d1-c8008ebe9657" containerName="revoke-keys"
Oct 30 01:16:35.513824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.513823    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a35d71-6e8d-4846-b7d1-c8008ebe9657" containerName="revoke-keys"
Oct 30 01:16:35.514044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.513853    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jc7t\" (UniqueName: \"kubernetes.io/projected/07a35d71-6e8d-4846-b7d1-c8008ebe9657-kube-api-access-6jc7t\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:35.514044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.513882    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a35d71-6e8d-4846-b7d1-c8008ebe9657-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:35.514156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.514114    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a35d71-6e8d-4846-b7d1-c8008ebe9657" containerName="revoke-keys"
Oct 30 01:16:35.518393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.518370    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.538801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.538766    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"]
Oct 30 01:16:35.615252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.615218    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.615577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.615532    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.615805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.615783    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccsb7\" (UniqueName: \"kubernetes.io/projected/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-kube-api-access-ccsb7\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.717447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.717412    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccsb7\" (UniqueName: \"kubernetes.io/projected/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-kube-api-access-ccsb7\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.717666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.717472    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.717666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.717644    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.718190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.718166    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.720124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.720095    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.748393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.748364    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccsb7\" (UniqueName: \"kubernetes.io/projected/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-kube-api-access-ccsb7\") pod \"maas-api-revoke-sub-1fca894e-q85q5\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:35.838670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:35.838632    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:36.076630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:36.076550    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"
Oct 30 01:16:36.076786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:36.076537    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7" event={"ID":"07a35d71-6e8d-4846-b7d1-c8008ebe9657","Type":"ContainerDied","Data":"1fcb0b9663306dc69e10e0bf5d5e41e24ecd657f62660d2a572ccbef8c8e6275"}
Oct 30 01:16:36.076786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:36.076715    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fcb0b9663306dc69e10e0bf5d5e41e24ecd657f62660d2a572ccbef8c8e6275"
Oct 30 01:16:36.155802 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:36.155766    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7537d8_e5c4_454a_b10c_a7c035d1cb22.slice/crio-a74849cf633d3e8562d152014d09e559ef93c16da782f6bfb5d019c13165fe13 WatchSource:0}: Error finding container a74849cf633d3e8562d152014d09e559ef93c16da782f6bfb5d019c13165fe13: Status 404 returned error can't find the container with id a74849cf633d3e8562d152014d09e559ef93c16da782f6bfb5d019c13165fe13
Oct 30 01:16:36.156235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:36.156207    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"]
Oct 30 01:16:37.088277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:37.088181    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5" event={"ID":"cf7537d8-e5c4-454a-b10c-a7c035d1cb22","Type":"ContainerStarted","Data":"003c2092bd8b625fe16e0361a7d83299bce2036960d8b0e8e3bcfe4273dce483"}
Oct 30 01:16:37.088277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:37.088241    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5" event={"ID":"cf7537d8-e5c4-454a-b10c-a7c035d1cb22","Type":"ContainerStarted","Data":"a74849cf633d3e8562d152014d09e559ef93c16da782f6bfb5d019c13165fe13"}
Oct 30 01:16:37.139127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:37.139061    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5" podStartSLOduration=2.139044764 podStartE2EDuration="2.139044764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:37.125760461 +0000 UTC m=+2038.673450956" watchObservedRunningTime="2026-10-30 01:16:37.139044764 +0000 UTC m=+2038.686735260"
Oct 30 01:16:38.136193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:38.136024    5104 generic.go:356] "Generic (PLEG): container finished" podID="cf7537d8-e5c4-454a-b10c-a7c035d1cb22" containerID="003c2092bd8b625fe16e0361a7d83299bce2036960d8b0e8e3bcfe4273dce483" exitCode=0
Oct 30 01:16:38.136695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:38.136272    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5" event={"ID":"cf7537d8-e5c4-454a-b10c-a7c035d1cb22","Type":"ContainerDied","Data":"003c2092bd8b625fe16e0361a7d83299bce2036960d8b0e8e3bcfe4273dce483"}
Oct 30 01:16:39.116429 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:16:39.116390    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:16:39.387616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.387552    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:39.437880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.437841    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-xks-ca\") pod \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") "
Oct 30 01:16:39.438090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.437912    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-openshift-ca\") pod \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") "
Oct 30 01:16:39.438090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.438072    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-kube-api-access-ccsb7\" (UniqueName: \"kubernetes.io/projected/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-kube-api-access-ccsb7\") pod \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\" (UID: \"cf7537d8-e5c4-454a-b10c-a7c035d1cb22\") "
Oct 30 01:16:39.438813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.438762    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-openshift-ca" pod "cf7537d8-e5c4-454a-b10c-a7c035d1cb22" (UID: "cf7537d8-e5c4-454a-b10c-a7c035d1cb22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:39.441017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.440987    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-xks-ca" pod "cf7537d8-e5c4-454a-b10c-a7c035d1cb22" (UID: "cf7537d8-e5c4-454a-b10c-a7c035d1cb22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:39.442081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.442057    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-kube-api-access-ccsb7" pod "cf7537d8-e5c4-454a-b10c-a7c035d1cb22" (UID: "cf7537d8-e5c4-454a-b10c-a7c035d1cb22"). InnerVolumeSpecName "kube-api-access-ccsb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:39.540383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.540346    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:39.540383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.540383    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:39.540616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:39.540395    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccsb7\" (UniqueName: \"kubernetes.io/projected/cf7537d8-e5c4-454a-b10c-a7c035d1cb22-kube-api-access-ccsb7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:40.158887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:40.158849    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"
Oct 30 01:16:40.158887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:40.158878    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5" event={"ID":"cf7537d8-e5c4-454a-b10c-a7c035d1cb22","Type":"ContainerDied","Data":"a74849cf633d3e8562d152014d09e559ef93c16da782f6bfb5d019c13165fe13"}
Oct 30 01:16:40.159153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:40.158928    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a74849cf633d3e8562d152014d09e559ef93c16da782f6bfb5d019c13165fe13"
Oct 30 01:16:41.155761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.155720    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"]
Oct 30 01:16:41.173363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.173324    5104 cpu_manager.go:414] "containerMap: removing container" podUID="cf7537d8-e5c4-454a-b10c-a7c035d1cb22" containerName="revoke-keys"
Oct 30 01:16:41.173632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.173613    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7537d8-e5c4-454a-b10c-a7c035d1cb22" containerName="revoke-keys"
Oct 30 01:16:41.174064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.174048    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7537d8-e5c4-454a-b10c-a7c035d1cb22" containerName="revoke-keys"
Oct 30 01:16:41.192804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.192770    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.241653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.234812    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"]
Oct 30 01:16:41.334953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.297826    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.376481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.356926    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzmm\" (UniqueName: \"kubernetes.io/projected/1cbe8731-8b97-4e3d-a673-85b5abe7df28-kube-api-access-8fzmm\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.379579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.376891    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.402488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.356273    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:41.415450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.356345    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:41.415733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.356123    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:16:41.415860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.356187    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:41.501891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.501855    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzmm\" (UniqueName: \"kubernetes.io/projected/1cbe8731-8b97-4e3d-a673-85b5abe7df28-kube-api-access-8fzmm\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.502286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.502257    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.502608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.502593    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.503765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.503745    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.507774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.507752    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.531706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.531668    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzmm\" (UniqueName: \"kubernetes.io/projected/1cbe8731-8b97-4e3d-a673-85b5abe7df28-kube-api-access-8fzmm\") pod \"maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.658046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.657999    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:41.826968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.826927    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"]
Oct 30 01:16:41.833349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.833324    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:41.858330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.857905    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"]
Oct 30 01:16:41.909700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.909657    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:41.910006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.909977    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:41.910502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.910427    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kp7\" (UniqueName: \"kubernetes.io/projected/f894045c-f5a9-4272-9cba-bb7f6156fc19-kube-api-access-b9kp7\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:41.973762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:41.973730    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"]
Oct 30 01:16:41.973906 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:41.973788    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbe8731_8b97_4e3d_a673_85b5abe7df28.slice/crio-b164ded7612e31360495bc36e940ae01ea8d31497cb4ec94b9accf016b4b659b WatchSource:0}: Error finding container b164ded7612e31360495bc36e940ae01ea8d31497cb4ec94b9accf016b4b659b: Status 404 returned error can't find the container with id b164ded7612e31360495bc36e940ae01ea8d31497cb4ec94b9accf016b4b659b
Oct 30 01:16:42.013024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.012984    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:42.013295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.013273    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kp7\" (UniqueName: \"kubernetes.io/projected/f894045c-f5a9-4272-9cba-bb7f6156fc19-kube-api-access-b9kp7\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:42.013411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.013342    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:42.014016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.013989    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:42.015532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.015513    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:42.032254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.032220    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kp7\" (UniqueName: \"kubernetes.io/projected/f894045c-f5a9-4272-9cba-bb7f6156fc19-kube-api-access-b9kp7\") pod \"maas-api-revoke-sub-4cd9d8d0-66wqr\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:42.153896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.153855    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:42.387436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.387381    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c" event={"ID":"1cbe8731-8b97-4e3d-a673-85b5abe7df28","Type":"ContainerStarted","Data":"b164ded7612e31360495bc36e940ae01ea8d31497cb4ec94b9accf016b4b659b"}
Oct 30 01:16:42.479432 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:42.479358    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf894045c_f5a9_4272_9cba_bb7f6156fc19.slice/crio-d03eb2f93c857c24f8c66140e5b572f7d9065c24f1242b09f91ed4f64e7f8611 WatchSource:0}: Error finding container d03eb2f93c857c24f8c66140e5b572f7d9065c24f1242b09f91ed4f64e7f8611: Status 404 returned error can't find the container with id d03eb2f93c857c24f8c66140e5b572f7d9065c24f1242b09f91ed4f64e7f8611
Oct 30 01:16:42.479720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:42.479684    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"]
Oct 30 01:16:43.402814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:43.402701    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr" event={"ID":"f894045c-f5a9-4272-9cba-bb7f6156fc19","Type":"ContainerStarted","Data":"a978b0d2be51c5c1d40b966be13a12219e6b1879e218848b7f07f5f6d0d20d6e"}
Oct 30 01:16:43.402814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:43.402811    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr" event={"ID":"f894045c-f5a9-4272-9cba-bb7f6156fc19","Type":"ContainerStarted","Data":"d03eb2f93c857c24f8c66140e5b572f7d9065c24f1242b09f91ed4f64e7f8611"}
Oct 30 01:16:43.404989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:43.404955    5104 generic.go:356] "Generic (PLEG): container finished" podID="1cbe8731-8b97-4e3d-a673-85b5abe7df28" containerID="36aaa391b42240784b85ff80f26b306b6d9f7a06ea1757be8ed2e187bbd20dcb" exitCode=0
Oct 30 01:16:43.405139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:43.405026    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c" event={"ID":"1cbe8731-8b97-4e3d-a673-85b5abe7df28","Type":"ContainerDied","Data":"36aaa391b42240784b85ff80f26b306b6d9f7a06ea1757be8ed2e187bbd20dcb"}
Oct 30 01:16:43.447323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:43.447252    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr" podStartSLOduration=2.447233432 podStartE2EDuration="2.447233432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:43.443516591 +0000 UTC m=+2044.991207088" watchObservedRunningTime="2026-10-30 01:16:43.447233432 +0000 UTC m=+2044.994923962"
Oct 30 01:16:44.418426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.418388    5104 generic.go:356] "Generic (PLEG): container finished" podID="f894045c-f5a9-4272-9cba-bb7f6156fc19" containerID="a978b0d2be51c5c1d40b966be13a12219e6b1879e218848b7f07f5f6d0d20d6e" exitCode=0
Oct 30 01:16:44.418933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.418485    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr" event={"ID":"f894045c-f5a9-4272-9cba-bb7f6156fc19","Type":"ContainerDied","Data":"a978b0d2be51c5c1d40b966be13a12219e6b1879e218848b7f07f5f6d0d20d6e"}
Oct 30 01:16:44.662390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.662356    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:44.675521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.675493    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cbe8731-8b97-4e3d-a673-85b5abe7df28-kube-api-access-8fzmm\" (UniqueName: \"kubernetes.io/projected/1cbe8731-8b97-4e3d-a673-85b5abe7df28-kube-api-access-8fzmm\") pod \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") "
Oct 30 01:16:44.675773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.675753    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-xks-ca\") pod \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") "
Oct 30 01:16:44.675953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.675932    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-openshift-ca\") pod \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\" (UID: \"1cbe8731-8b97-4e3d-a673-85b5abe7df28\") "
Oct 30 01:16:44.676323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.676294    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-openshift-ca" pod "1cbe8731-8b97-4e3d-a673-85b5abe7df28" (UID: "1cbe8731-8b97-4e3d-a673-85b5abe7df28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:44.677237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.677217    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:44.678626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.678600    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-xks-ca" pod "1cbe8731-8b97-4e3d-a673-85b5abe7df28" (UID: "1cbe8731-8b97-4e3d-a673-85b5abe7df28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:44.679197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.679111    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbe8731-8b97-4e3d-a673-85b5abe7df28-kube-api-access-8fzmm" pod "1cbe8731-8b97-4e3d-a673-85b5abe7df28" (UID: "1cbe8731-8b97-4e3d-a673-85b5abe7df28"). InnerVolumeSpecName "kube-api-access-8fzmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:44.782347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.782306    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbe8731-8b97-4e3d-a673-85b5abe7df28-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:44.782347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:44.782349    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fzmm\" (UniqueName: \"kubernetes.io/projected/1cbe8731-8b97-4e3d-a673-85b5abe7df28-kube-api-access-8fzmm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:45.431809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.431771    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"
Oct 30 01:16:45.431809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.431782    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c" event={"ID":"1cbe8731-8b97-4e3d-a673-85b5abe7df28","Type":"ContainerDied","Data":"b164ded7612e31360495bc36e940ae01ea8d31497cb4ec94b9accf016b4b659b"}
Oct 30 01:16:45.432363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.431906    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b164ded7612e31360495bc36e940ae01ea8d31497cb4ec94b9accf016b4b659b"
Oct 30 01:16:45.686617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.686535    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:45.802104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.802057    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-openshift-ca\") pod \"f894045c-f5a9-4272-9cba-bb7f6156fc19\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") "
Oct 30 01:16:45.802323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.802221    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f894045c-f5a9-4272-9cba-bb7f6156fc19-kube-api-access-b9kp7\" (UniqueName: \"kubernetes.io/projected/f894045c-f5a9-4272-9cba-bb7f6156fc19-kube-api-access-b9kp7\") pod \"f894045c-f5a9-4272-9cba-bb7f6156fc19\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") "
Oct 30 01:16:45.802323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.802294    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-xks-ca\") pod \"f894045c-f5a9-4272-9cba-bb7f6156fc19\" (UID: \"f894045c-f5a9-4272-9cba-bb7f6156fc19\") "
Oct 30 01:16:45.803631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.803593    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-openshift-ca" pod "f894045c-f5a9-4272-9cba-bb7f6156fc19" (UID: "f894045c-f5a9-4272-9cba-bb7f6156fc19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:45.808123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.806229    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f894045c-f5a9-4272-9cba-bb7f6156fc19-kube-api-access-b9kp7" pod "f894045c-f5a9-4272-9cba-bb7f6156fc19" (UID: "f894045c-f5a9-4272-9cba-bb7f6156fc19"). InnerVolumeSpecName "kube-api-access-b9kp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:45.821724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.808550    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-xks-ca" pod "f894045c-f5a9-4272-9cba-bb7f6156fc19" (UID: "f894045c-f5a9-4272-9cba-bb7f6156fc19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:45.840548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.840514    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"]
Oct 30 01:16:45.841661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.841635    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f894045c-f5a9-4272-9cba-bb7f6156fc19" containerName="revoke-keys"
Oct 30 01:16:45.841661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.841661    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f894045c-f5a9-4272-9cba-bb7f6156fc19" containerName="revoke-keys"
Oct 30 01:16:45.841854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.841777    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1cbe8731-8b97-4e3d-a673-85b5abe7df28" containerName="revoke-keys"
Oct 30 01:16:45.841854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.841786    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbe8731-8b97-4e3d-a673-85b5abe7df28" containerName="revoke-keys"
Oct 30 01:16:45.842060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.842043    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbe8731-8b97-4e3d-a673-85b5abe7df28" containerName="revoke-keys"
Oct 30 01:16:45.842130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.842079    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f894045c-f5a9-4272-9cba-bb7f6156fc19" containerName="revoke-keys"
Oct 30 01:16:45.846802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.846778    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:45.869168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.869133    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"]
Oct 30 01:16:45.905481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.905219    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:45.905481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.905431    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjt2l\" (UniqueName: \"kubernetes.io/projected/850c8723-4307-44b1-b6e0-91e76568f158-kube-api-access-hjt2l\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:45.905759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.905509    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:45.907858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.905868    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:45.907858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.905888    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9kp7\" (UniqueName: \"kubernetes.io/projected/f894045c-f5a9-4272-9cba-bb7f6156fc19-kube-api-access-b9kp7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:45.907858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:45.905903    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f894045c-f5a9-4272-9cba-bb7f6156fc19-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.007274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.007176    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjt2l\" (UniqueName: \"kubernetes.io/projected/850c8723-4307-44b1-b6e0-91e76568f158-kube-api-access-hjt2l\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:46.007274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.007242    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:46.007557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.007483    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:46.008385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.008314    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:46.010004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.009977    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:46.048488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.027178    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjt2l\" (UniqueName: \"kubernetes.io/projected/850c8723-4307-44b1-b6e0-91e76568f158-kube-api-access-hjt2l\") pod \"maas-api-revoke-sub-f821dd22-mblzg\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:46.177428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.177388    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:46.503573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.503518    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr" event={"ID":"f894045c-f5a9-4272-9cba-bb7f6156fc19","Type":"ContainerDied","Data":"d03eb2f93c857c24f8c66140e5b572f7d9065c24f1242b09f91ed4f64e7f8611"}
Oct 30 01:16:46.504057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.503598    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d03eb2f93c857c24f8c66140e5b572f7d9065c24f1242b09f91ed4f64e7f8611"
Oct 30 01:16:46.504057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.503686    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"
Oct 30 01:16:46.663127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.663092    5104 scope.go:122] "RemoveContainer" containerID="f4e4eb4d8964614a5c2c83cbb8b45db66c58ba71df4030abf95b9c7894678f05"
Oct 30 01:16:46.677924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.677876    5104 scope.go:122] "RemoveContainer" containerID="2c1372ae8b5f0483c328d6e1841592f4b0e8004e993a874a63733637bcf8e7fd"
Oct 30 01:16:46.698849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:46.698799    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"]
Oct 30 01:16:47.517841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:47.517735    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg" event={"ID":"850c8723-4307-44b1-b6e0-91e76568f158","Type":"ContainerStarted","Data":"522df8d322cfc465e7b1f6f1d0346ac9572e8e00c848648de86b582bda8dfc98"}
Oct 30 01:16:47.517841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:47.517811    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg" event={"ID":"850c8723-4307-44b1-b6e0-91e76568f158","Type":"ContainerStarted","Data":"b2779cd20c26f06146a4a8fc7013bb061256e5573bb66559bb6566c159053323"}
Oct 30 01:16:47.565511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:47.565442    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg" podStartSLOduration=2.5654274470000002 podStartE2EDuration="2.565427447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:47.563866423 +0000 UTC m=+2049.111556919" watchObservedRunningTime="2026-10-30 01:16:47.565427447 +0000 UTC m=+2049.113117944"
Oct 30 01:16:48.529834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:48.529796    5104 generic.go:356] "Generic (PLEG): container finished" podID="850c8723-4307-44b1-b6e0-91e76568f158" containerID="522df8d322cfc465e7b1f6f1d0346ac9572e8e00c848648de86b582bda8dfc98" exitCode=0
Oct 30 01:16:48.530234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:48.529842    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg" event={"ID":"850c8723-4307-44b1-b6e0-91e76568f158","Type":"ContainerDied","Data":"522df8d322cfc465e7b1f6f1d0346ac9572e8e00c848648de86b582bda8dfc98"}
Oct 30 01:16:49.799608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.799521    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:49.951497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.951466    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850c8723-4307-44b1-b6e0-91e76568f158-kube-api-access-hjt2l\" (UniqueName: \"kubernetes.io/projected/850c8723-4307-44b1-b6e0-91e76568f158-kube-api-access-hjt2l\") pod \"850c8723-4307-44b1-b6e0-91e76568f158\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") "
Oct 30 01:16:49.951669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.951526    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-openshift-ca\") pod \"850c8723-4307-44b1-b6e0-91e76568f158\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") "
Oct 30 01:16:49.951669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.951618    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-xks-ca\") pod \"850c8723-4307-44b1-b6e0-91e76568f158\" (UID: \"850c8723-4307-44b1-b6e0-91e76568f158\") "
Oct 30 01:16:49.952033 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.951994    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-openshift-ca" pod "850c8723-4307-44b1-b6e0-91e76568f158" (UID: "850c8723-4307-44b1-b6e0-91e76568f158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:49.952703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.952676    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:49.953831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.953803    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-xks-ca" pod "850c8723-4307-44b1-b6e0-91e76568f158" (UID: "850c8723-4307-44b1-b6e0-91e76568f158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:49.953956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:49.953893    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850c8723-4307-44b1-b6e0-91e76568f158-kube-api-access-hjt2l" pod "850c8723-4307-44b1-b6e0-91e76568f158" (UID: "850c8723-4307-44b1-b6e0-91e76568f158"). InnerVolumeSpecName "kube-api-access-hjt2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:50.054495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:50.054396    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjt2l\" (UniqueName: \"kubernetes.io/projected/850c8723-4307-44b1-b6e0-91e76568f158-kube-api-access-hjt2l\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:50.054495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:50.054426    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850c8723-4307-44b1-b6e0-91e76568f158-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:50.549114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:50.549081    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"
Oct 30 01:16:50.549114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:50.549105    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg" event={"ID":"850c8723-4307-44b1-b6e0-91e76568f158","Type":"ContainerDied","Data":"b2779cd20c26f06146a4a8fc7013bb061256e5573bb66559bb6566c159053323"}
Oct 30 01:16:50.549376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:50.549158    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2779cd20c26f06146a4a8fc7013bb061256e5573bb66559bb6566c159053323"
Oct 30 01:16:51.654984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:51.654950    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"]
Oct 30 01:16:51.699602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:51.699564    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"]
Oct 30 01:16:53.285529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.285488    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"]
Oct 30 01:16:53.286967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.286946    5104 cpu_manager.go:414] "containerMap: removing container" podUID="850c8723-4307-44b1-b6e0-91e76568f158" containerName="revoke-keys"
Oct 30 01:16:53.287055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.286970    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850c8723-4307-44b1-b6e0-91e76568f158" containerName="revoke-keys"
Oct 30 01:16:53.287215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.287203    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="850c8723-4307-44b1-b6e0-91e76568f158" containerName="revoke-keys"
Oct 30 01:16:53.292474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.292428    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.306356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.306327    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"]
Oct 30 01:16:53.310257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.310232    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:16:53.310257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.310241    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:16:53.310491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.310256    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:53.315647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.315617    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:16:53.408929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.408873    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqd4l\" (UniqueName: \"kubernetes.io/projected/55f6822e-b4ee-48ec-9d64-5972ee382a68-kube-api-access-nqd4l\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.409144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.409125    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.409304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.409288    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.511034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.510998    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.511231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.511195    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.511384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.511365    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqd4l\" (UniqueName: \"kubernetes.io/projected/55f6822e-b4ee-48ec-9d64-5972ee382a68-kube-api-access-nqd4l\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.511879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.511854    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.513682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.513659    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.529958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.529930    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqd4l\" (UniqueName: \"kubernetes.io/projected/55f6822e-b4ee-48ec-9d64-5972ee382a68-kube-api-access-nqd4l\") pod \"maas-api-revoke-sub-fb5c7fe0-hwllp\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.612812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.612717    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:53.930775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:53.930743    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"]
Oct 30 01:16:54.656948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.656863    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"]
Oct 30 01:16:54.662237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.662212    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.684183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.684136    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"]
Oct 30 01:16:54.742487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.739946    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp" event={"ID":"55f6822e-b4ee-48ec-9d64-5972ee382a68","Type":"ContainerStarted","Data":"81f3f42c6aa050aedfeb0b51feefe2ea40295398cb5fd93d9791679e189a16dd"}
Oct 30 01:16:54.742487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.740025    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp" event={"ID":"55f6822e-b4ee-48ec-9d64-5972ee382a68","Type":"ContainerStarted","Data":"255c42ed3ad216144db8061a51ed19278aa938b444f727d917a55d3c54863be3"}
Oct 30 01:16:54.747520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.743219    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.747520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.743553    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.747520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.743795    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqn9j\" (UniqueName: \"kubernetes.io/projected/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-kube-api-access-lqn9j\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.809522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.802768    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp" podStartSLOduration=1.802747536 podStartE2EDuration="1.802747536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:54.80129182 +0000 UTC m=+2056.348982353" watchObservedRunningTime="2026-10-30 01:16:54.802747536 +0000 UTC m=+2056.350438032"
Oct 30 01:16:54.847493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.845960    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.847493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.846181    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.847493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.846389    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn9j\" (UniqueName: \"kubernetes.io/projected/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-kube-api-access-lqn9j\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.847493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.847351    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.850441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.850281    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.866922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.866893    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqn9j\" (UniqueName: \"kubernetes.io/projected/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-kube-api-access-lqn9j\") pod \"maas-api-revoke-sub-4b9066fe-9r86c\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:54.982205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:54.982172    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:55.304763 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:16:55.304724    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6265cd_38aa_4257_bb7a_8f2e11fd8fa4.slice/crio-8f82f966090e55d717ebb3f75f7993e7ffe209e21f758861b85a287efa3184aa WatchSource:0}: Error finding container 8f82f966090e55d717ebb3f75f7993e7ffe209e21f758861b85a287efa3184aa: Status 404 returned error can't find the container with id 8f82f966090e55d717ebb3f75f7993e7ffe209e21f758861b85a287efa3184aa
Oct 30 01:16:55.304915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:55.304844    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"]
Oct 30 01:16:55.751130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:55.751093    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c" event={"ID":"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4","Type":"ContainerStarted","Data":"8f82f966090e55d717ebb3f75f7993e7ffe209e21f758861b85a287efa3184aa"}
Oct 30 01:16:55.752956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:55.752921    5104 generic.go:356] "Generic (PLEG): container finished" podID="55f6822e-b4ee-48ec-9d64-5972ee382a68" containerID="81f3f42c6aa050aedfeb0b51feefe2ea40295398cb5fd93d9791679e189a16dd" exitCode=0
Oct 30 01:16:55.753127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:55.752964    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp" event={"ID":"55f6822e-b4ee-48ec-9d64-5972ee382a68","Type":"ContainerDied","Data":"81f3f42c6aa050aedfeb0b51feefe2ea40295398cb5fd93d9791679e189a16dd"}
Oct 30 01:16:56.605644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:56.605600    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:16:56.655922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:56.655837    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" podUID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" containerName="payload-processing" containerID="cri-o://758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43" gracePeriod=25
Oct 30 01:16:56.700544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:56.700503    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" podUID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" containerName="payload-pre-processing" containerID="cri-o://efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065" gracePeriod=25
Oct 30 01:16:56.765212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:56.765180    5104 generic.go:356] "Generic (PLEG): container finished" podID="4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" containerID="d9ddbe3e474f7ff6248139a37157fd3b501e77fa60290f9947b20eaf78a4d348" exitCode=0
Oct 30 01:16:56.765669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:56.765249    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c" event={"ID":"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4","Type":"ContainerDied","Data":"d9ddbe3e474f7ff6248139a37157fd3b501e77fa60290f9947b20eaf78a4d348"}
Oct 30 01:16:56.980658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:56.980632    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:57.092099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.092060    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-openshift-ca\") pod \"55f6822e-b4ee-48ec-9d64-5972ee382a68\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") "
Oct 30 01:16:57.092341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.092319    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-xks-ca\") pod \"55f6822e-b4ee-48ec-9d64-5972ee382a68\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") "
Oct 30 01:16:57.092433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.092360    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f6822e-b4ee-48ec-9d64-5972ee382a68-kube-api-access-nqd4l\" (UniqueName: \"kubernetes.io/projected/55f6822e-b4ee-48ec-9d64-5972ee382a68-kube-api-access-nqd4l\") pod \"55f6822e-b4ee-48ec-9d64-5972ee382a68\" (UID: \"55f6822e-b4ee-48ec-9d64-5972ee382a68\") "
Oct 30 01:16:57.092560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.092531    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-openshift-ca" pod "55f6822e-b4ee-48ec-9d64-5972ee382a68" (UID: "55f6822e-b4ee-48ec-9d64-5972ee382a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:57.093270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.093242    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:57.095349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.095320    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f6822e-b4ee-48ec-9d64-5972ee382a68-kube-api-access-nqd4l" pod "55f6822e-b4ee-48ec-9d64-5972ee382a68" (UID: "55f6822e-b4ee-48ec-9d64-5972ee382a68"). InnerVolumeSpecName "kube-api-access-nqd4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:57.098603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.098574    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-xks-ca" pod "55f6822e-b4ee-48ec-9d64-5972ee382a68" (UID: "55f6822e-b4ee-48ec-9d64-5972ee382a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:57.195186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.195099    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f6822e-b4ee-48ec-9d64-5972ee382a68-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:57.195186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.195131    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqd4l\" (UniqueName: \"kubernetes.io/projected/55f6822e-b4ee-48ec-9d64-5972ee382a68-kube-api-access-nqd4l\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:57.843482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.829021    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"
Oct 30 01:16:57.844422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.843620    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp" event={"ID":"55f6822e-b4ee-48ec-9d64-5972ee382a68","Type":"ContainerDied","Data":"255c42ed3ad216144db8061a51ed19278aa938b444f727d917a55d3c54863be3"}
Oct 30 01:16:57.844422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:57.843744    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="255c42ed3ad216144db8061a51ed19278aa938b444f727d917a55d3c54863be3"
Oct 30 01:16:58.480678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.478009    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:58.613714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.613628    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-xks-ca\") pod \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") "
Oct 30 01:16:58.613965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.613940    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-kube-api-access-lqn9j\" (UniqueName: \"kubernetes.io/projected/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-kube-api-access-lqn9j\") pod \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") "
Oct 30 01:16:58.614339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.614317    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-openshift-ca\") pod \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\" (UID: \"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4\") "
Oct 30 01:16:58.616829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.616799    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-openshift-ca" pod "4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" (UID: "4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:58.758480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.754242    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:58.791537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.788408    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-xks-ca" pod "4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" (UID: "4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:58.895591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.895501    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:58.976626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:58.963766    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-kube-api-access-lqn9j" pod "4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" (UID: "4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4"). InnerVolumeSpecName "kube-api-access-lqn9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:59.049481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:59.024110    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqn9j\" (UniqueName: \"kubernetes.io/projected/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4-kube-api-access-lqn9j\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:16:59.053753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:59.026856    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c" event={"ID":"4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4","Type":"ContainerDied","Data":"8f82f966090e55d717ebb3f75f7993e7ffe209e21f758861b85a287efa3184aa"}
Oct 30 01:16:59.054009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:59.053985    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f82f966090e55d717ebb3f75f7993e7ffe209e21f758861b85a287efa3184aa"
Oct 30 01:16:59.054137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:16:59.026934    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"
Oct 30 01:16:59.315606 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:16:59.310902    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6265cd_38aa_4257_bb7a_8f2e11fd8fa4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:17:04.759649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:04.759604    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" podUID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:17:05.584953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:05.584910    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"]
Oct 30 01:17:05.850528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:05.849788    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" podUID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:17:06.261825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.261785    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"]
Oct 30 01:17:06.262997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.262972    5104 cpu_manager.go:414] "containerMap: removing container" podUID="55f6822e-b4ee-48ec-9d64-5972ee382a68" containerName="revoke-keys"
Oct 30 01:17:06.262997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.262994    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f6822e-b4ee-48ec-9d64-5972ee382a68" containerName="revoke-keys"
Oct 30 01:17:06.263173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.263057    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" containerName="revoke-keys"
Oct 30 01:17:06.263173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.263066    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" containerName="revoke-keys"
Oct 30 01:17:06.263312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.263294    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" containerName="revoke-keys"
Oct 30 01:17:06.263352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.263323    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f6822e-b4ee-48ec-9d64-5972ee382a68" containerName="revoke-keys"
Oct 30 01:17:06.267101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.267082    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.287487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.287275    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:06.287487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.287429    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:06.289279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.289245    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:17:06.289537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.289517    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:06.331483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.307727    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"]
Oct 30 01:17:06.414487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.404046    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.414487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.404219    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhnwl\" (UniqueName: \"kubernetes.io/projected/6d148f0d-2186-472c-81df-0d88cef41a48-kube-api-access-mhnwl\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.414487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.404499    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.506563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.506525    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.506753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.506655    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.506804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.506758    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhnwl\" (UniqueName: \"kubernetes.io/projected/6d148f0d-2186-472c-81df-0d88cef41a48-kube-api-access-mhnwl\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.507392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.507368    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.509050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.509027    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.525761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.525701    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhnwl\" (UniqueName: \"kubernetes.io/projected/6d148f0d-2186-472c-81df-0d88cef41a48-kube-api-access-mhnwl\") pod \"maas-api-revoke-sub-c8cea6f9-k2x87\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.595907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.595873    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:06.928526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:06.910851    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"]
Oct 30 01:17:06.928526 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:17:06.925380    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d148f0d_2186_472c_81df_0d88cef41a48.slice/crio-57147664b6f771f2697122321e5f4b40fba9d476770c5680c3c5f697ca203144 WatchSource:0}: Error finding container 57147664b6f771f2697122321e5f4b40fba9d476770c5680c3c5f697ca203144: Status 404 returned error can't find the container with id 57147664b6f771f2697122321e5f4b40fba9d476770c5680c3c5f697ca203144
Oct 30 01:17:07.110982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:07.110929    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87" event={"ID":"6d148f0d-2186-472c-81df-0d88cef41a48","Type":"ContainerStarted","Data":"57147664b6f771f2697122321e5f4b40fba9d476770c5680c3c5f697ca203144"}
Oct 30 01:17:07.199935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:07.199899    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"]
Oct 30 01:17:08.123150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:08.123119    5104 generic.go:356] "Generic (PLEG): container finished" podID="6d148f0d-2186-472c-81df-0d88cef41a48" containerID="48e7b0ef74b6207c8965d42724eb60d9d719d935dac0beedf63f00b48f08d249" exitCode=0
Oct 30 01:17:08.123565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:08.123211    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87" event={"ID":"6d148f0d-2186-472c-81df-0d88cef41a48","Type":"ContainerDied","Data":"48e7b0ef74b6207c8965d42724eb60d9d719d935dac0beedf63f00b48f08d249"}
Oct 30 01:17:09.106980 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:09.106940    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:17:09.361373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.361300    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:09.480514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.480475    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-xks-ca\") pod \"6d148f0d-2186-472c-81df-0d88cef41a48\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") "
Oct 30 01:17:09.480719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.480545    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d148f0d-2186-472c-81df-0d88cef41a48-kube-api-access-mhnwl\" (UniqueName: \"kubernetes.io/projected/6d148f0d-2186-472c-81df-0d88cef41a48-kube-api-access-mhnwl\") pod \"6d148f0d-2186-472c-81df-0d88cef41a48\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") "
Oct 30 01:17:09.480807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.480749    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-openshift-ca\") pod \"6d148f0d-2186-472c-81df-0d88cef41a48\" (UID: \"6d148f0d-2186-472c-81df-0d88cef41a48\") "
Oct 30 01:17:09.481328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.481278    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-openshift-ca" pod "6d148f0d-2186-472c-81df-0d88cef41a48" (UID: "6d148f0d-2186-472c-81df-0d88cef41a48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:09.481980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.481954    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:09.483010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.482986    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-xks-ca" pod "6d148f0d-2186-472c-81df-0d88cef41a48" (UID: "6d148f0d-2186-472c-81df-0d88cef41a48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:09.483551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.483523    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d148f0d-2186-472c-81df-0d88cef41a48-kube-api-access-mhnwl" pod "6d148f0d-2186-472c-81df-0d88cef41a48" (UID: "6d148f0d-2186-472c-81df-0d88cef41a48"). InnerVolumeSpecName "kube-api-access-mhnwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:09.607477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.596775    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d148f0d-2186-472c-81df-0d88cef41a48-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:09.607477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:09.596816    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhnwl\" (UniqueName: \"kubernetes.io/projected/6d148f0d-2186-472c-81df-0d88cef41a48-kube-api-access-mhnwl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:10.145760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:10.145728    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"
Oct 30 01:17:10.145970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:10.145769    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87" event={"ID":"6d148f0d-2186-472c-81df-0d88cef41a48","Type":"ContainerDied","Data":"57147664b6f771f2697122321e5f4b40fba9d476770c5680c3c5f697ca203144"}
Oct 30 01:17:10.145970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:10.145817    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57147664b6f771f2697122321e5f4b40fba9d476770c5680c3c5f697ca203144"
Oct 30 01:17:10.298476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:10.295745    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"]
Oct 30 01:17:14.774272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:14.773708    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" podUID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:17:15.818925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:15.818886    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" podUID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:17:17.067391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.067356    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:17:17.115349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.114254    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:17:17.125608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.125566    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"]
Oct 30 01:17:17.215532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.203178    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-kube-api-access-t86hn\" (UniqueName: \"kubernetes.io/projected/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-kube-api-access-t86hn\") pod \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") "
Oct 30 01:17:17.215532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.203583    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f33366-e2a3-4e2b-86a5-4924e65d3b69-kube-api-access-qzspv\" (UniqueName: \"kubernetes.io/projected/23f33366-e2a3-4e2b-86a5-4924e65d3b69-kube-api-access-qzspv\") pod \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") "
Oct 30 01:17:17.215532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.207544    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-kube-api-access-t86hn" pod "fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" (UID: "fa431ff4-1718-443e-ad8b-7a60ca7e7a3d"). InnerVolumeSpecName "kube-api-access-t86hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:17.215532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.210546    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f33366-e2a3-4e2b-86a5-4924e65d3b69-kube-api-access-qzspv" pod "23f33366-e2a3-4e2b-86a5-4924e65d3b69" (UID: "23f33366-e2a3-4e2b-86a5-4924e65d3b69"). InnerVolumeSpecName "kube-api-access-qzspv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:17.263483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.244123    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-plugins-config-volume\") pod \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\" (UID: \"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d\") "
Oct 30 01:17:17.263483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.244286    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f33366-e2a3-4e2b-86a5-4924e65d3b69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f33366-e2a3-4e2b-86a5-4924e65d3b69-plugins-config-volume\") pod \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\" (UID: \"23f33366-e2a3-4e2b-86a5-4924e65d3b69\") "
Oct 30 01:17:17.336501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.328729    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f33366-e2a3-4e2b-86a5-4924e65d3b69-plugins-config-volume" pod "23f33366-e2a3-4e2b-86a5-4924e65d3b69" (UID: "23f33366-e2a3-4e2b-86a5-4924e65d3b69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:17.336501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.329508    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzspv\" (UniqueName: \"kubernetes.io/projected/23f33366-e2a3-4e2b-86a5-4924e65d3b69-kube-api-access-qzspv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.336501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.329539    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f33366-e2a3-4e2b-86a5-4924e65d3b69-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.336501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.329553    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86hn\" (UniqueName: \"kubernetes.io/projected/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-kube-api-access-t86hn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.422475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.417646    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-plugins-config-volume" pod "fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" (UID: "fa431ff4-1718-443e-ad8b-7a60ca7e7a3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:17.445666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.445627    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.455435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.455403    5104 generic.go:356] "Generic (PLEG): container finished" podID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" containerID="efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065" exitCode=0
Oct 30 01:17:17.456052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.456032    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"
Oct 30 01:17:17.465431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.465062    5104 generic.go:356] "Generic (PLEG): container finished" podID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" containerID="758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43" exitCode=0
Oct 30 01:17:17.465431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.465173    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"
Oct 30 01:17:17.488327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.465117    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" event={"ID":"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d","Type":"ContainerDied","Data":"efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065"}
Oct 30 01:17:17.488730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.488372    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps" event={"ID":"fa431ff4-1718-443e-ad8b-7a60ca7e7a3d","Type":"ContainerDied","Data":"ee79cbea9c3b52886194c748482866dacb3b43cf2a78df27a4dd407749ffb9b5"}
Oct 30 01:17:17.488730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.488396    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" event={"ID":"23f33366-e2a3-4e2b-86a5-4924e65d3b69","Type":"ContainerDied","Data":"758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43"}
Oct 30 01:17:17.488730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.488410    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr" event={"ID":"23f33366-e2a3-4e2b-86a5-4924e65d3b69","Type":"ContainerDied","Data":"fa6e96c09a1ba11ebb9f051bd498598234a10587ed58e5bea8411477e0644c51"}
Oct 30 01:17:17.488730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.488430    5104 scope.go:122] "RemoveContainer" containerID="efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065"
Oct 30 01:17:17.505576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.505554    5104 scope.go:122] "RemoveContainer" containerID="efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065"
Oct 30 01:17:17.505972 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:17.505953    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065\": container with ID starting with efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065 not found: ID does not exist" containerID="efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065"
Oct 30 01:17:17.506066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.505982    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065"} err="failed to get container status \"efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065\": rpc error: code = NotFound desc = could not find container \"efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065\": container with ID starting with efa3f66c244d2020cc347d65a8a0c56eae8b35b6c50eeb2187b94fedd7162065 not found: ID does not exist"
Oct 30 01:17:17.506066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.506002    5104 scope.go:122] "RemoveContainer" containerID="758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43"
Oct 30 01:17:17.520424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.520399    5104 scope.go:122] "RemoveContainer" containerID="758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43"
Oct 30 01:17:17.520773 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:17.520750    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43\": container with ID starting with 758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43 not found: ID does not exist" containerID="758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43"
Oct 30 01:17:17.520873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.520782    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43"} err="failed to get container status \"758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43\": rpc error: code = NotFound desc = could not find container \"758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43\": container with ID starting with 758130a2d76ee22fe7e26dab97b39ed4189b7670b7d644a0e7d3d05d24e19c43 not found: ID does not exist"
Oct 30 01:17:17.535338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.535302    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"]
Oct 30 01:17:17.545600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.545569    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52f1cb-7f8d487558-cqcqr"]
Oct 30 01:17:17.582788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.578238    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"]
Oct 30 01:17:17.586817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:17.586783    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52f1cb-74c68749f-qx4ps"]
Oct 30 01:17:18.592044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:18.591987    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" path="/var/lib/kubelet/pods/23f33366-e2a3-4e2b-86a5-4924e65d3b69/volumes"
Oct 30 01:17:18.592980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:18.592955    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" path="/var/lib/kubelet/pods/fa431ff4-1718-443e-ad8b-7a60ca7e7a3d/volumes"
Oct 30 01:17:20.613774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:20.613707    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" podUID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" containerName="main" containerID="cri-o://59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f" gracePeriod=45
Oct 30 01:17:20.930514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:20.930484    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:17:21.026053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.025988    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tmp-dir\") pod \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") "
Oct 30 01:17:21.026274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026145    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-dshm\") pod \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") "
Oct 30 01:17:21.026367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026291    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-home\") pod \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") "
Oct 30 01:17:21.026367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026337    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tmp-dir" pod "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" (UID: "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.026542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026395    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-kube-api-access-nhhbd\" (UniqueName: \"kubernetes.io/projected/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-kube-api-access-nhhbd\") pod \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") "
Oct 30 01:17:21.026542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026508    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-model-cache\") pod \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") "
Oct 30 01:17:21.026680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026649    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-home" pod "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" (UID: "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.026744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026657    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tls-certs\") pod \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\" (UID: \"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa\") "
Oct 30 01:17:21.026845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.026811    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-model-cache" pod "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" (UID: "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.027687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.027659    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.027687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.027689    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.027920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.027711    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.029518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.029443    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-dshm" pod "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" (UID: "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.029647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.029634    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-kube-api-access-nhhbd" pod "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" (UID: "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa"). InnerVolumeSpecName "kube-api-access-nhhbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:21.029737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.029710    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tls-certs" pod "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" (UID: "f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:21.136493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.133546    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.136493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.133588    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhhbd\" (UniqueName: \"kubernetes.io/projected/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-kube-api-access-nhhbd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.136493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.133603    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.517351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.517310    5104 generic.go:356] "Generic (PLEG): container finished" podID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" containerID="59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f" exitCode=0
Oct 30 01:17:21.517351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.517354    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"
Oct 30 01:17:21.517644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.517366    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" event={"ID":"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa","Type":"ContainerDied","Data":"59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f"}
Oct 30 01:17:21.517644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.517410    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294" event={"ID":"f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa","Type":"ContainerDied","Data":"b039c7535584533cd1e6ce4057ccb1c745c61f2b0c1cd01ae47a573c9fe0e464"}
Oct 30 01:17:21.517644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.517431    5104 scope.go:122] "RemoveContainer" containerID="59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f"
Oct 30 01:17:21.536857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.536835    5104 scope.go:122] "RemoveContainer" containerID="59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f"
Oct 30 01:17:21.537217 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:21.537197    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f\": container with ID starting with 59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f not found: ID does not exist" containerID="59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f"
Oct 30 01:17:21.537309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.537223    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f"} err="failed to get container status \"59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f\": rpc error: code = NotFound desc = could not find container \"59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f\": container with ID starting with 59ac3c793aecfd0388459905c588395468155f62c36ad4ba22f9cf3d4746366f not found: ID does not exist"
Oct 30 01:17:21.569825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.569788    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"]
Oct 30 01:17:21.584651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:21.584613    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4dde48/test-model-4dde48-kserve-5f99d5bc77-cc294"]
Oct 30 01:17:22.223953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.223917    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main" containerID="cri-o://3c66a64312fe5aac1754bcb9db178c87006678a90cb6ec1324e057136d0509b1" gracePeriod=45
Oct 30 01:17:22.533573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.533485    5104 generic.go:356] "Generic (PLEG): container finished" podID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerID="3c66a64312fe5aac1754bcb9db178c87006678a90cb6ec1324e057136d0509b1" exitCode=0
Oct 30 01:17:22.533573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.533576    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" event={"ID":"b9c84a28-d678-4a81-a8f5-0ecf73662359","Type":"ContainerDied","Data":"3c66a64312fe5aac1754bcb9db178c87006678a90cb6ec1324e057136d0509b1"}
Oct 30 01:17:22.584919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.584879    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" path="/var/lib/kubelet/pods/f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa/volumes"
Oct 30 01:17:22.672824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.672793    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:17:22.766352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.766312    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c84a28-d678-4a81-a8f5-0ecf73662359-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c84a28-d678-4a81-a8f5-0ecf73662359-tls-certs\") pod \"b9c84a28-d678-4a81-a8f5-0ecf73662359\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") "
Oct 30 01:17:22.766574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.766435    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c84a28-d678-4a81-a8f5-0ecf73662359-kube-api-access-hl5wf\" (UniqueName: \"kubernetes.io/projected/b9c84a28-d678-4a81-a8f5-0ecf73662359-kube-api-access-hl5wf\") pod \"b9c84a28-d678-4a81-a8f5-0ecf73662359\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") "
Oct 30 01:17:22.766574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.766542    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-home\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-home\") pod \"b9c84a28-d678-4a81-a8f5-0ecf73662359\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") "
Oct 30 01:17:22.766704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.766598    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-dshm\") pod \"b9c84a28-d678-4a81-a8f5-0ecf73662359\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") "
Oct 30 01:17:22.766808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.766768    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-model-cache\") pod \"b9c84a28-d678-4a81-a8f5-0ecf73662359\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") "
Oct 30 01:17:22.766888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.766866    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-tmp-dir\") pod \"b9c84a28-d678-4a81-a8f5-0ecf73662359\" (UID: \"b9c84a28-d678-4a81-a8f5-0ecf73662359\") "
Oct 30 01:17:22.767121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.767064    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-model-cache" pod "b9c84a28-d678-4a81-a8f5-0ecf73662359" (UID: "b9c84a28-d678-4a81-a8f5-0ecf73662359"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:22.767195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.767143    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-home" pod "b9c84a28-d678-4a81-a8f5-0ecf73662359" (UID: "b9c84a28-d678-4a81-a8f5-0ecf73662359"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:22.767195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.767154    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-tmp-dir" pod "b9c84a28-d678-4a81-a8f5-0ecf73662359" (UID: "b9c84a28-d678-4a81-a8f5-0ecf73662359"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:22.767723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.767689    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:22.767723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.767722    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:22.767871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.767736    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:22.769942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.769911    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c84a28-d678-4a81-a8f5-0ecf73662359-tls-certs" pod "b9c84a28-d678-4a81-a8f5-0ecf73662359" (UID: "b9c84a28-d678-4a81-a8f5-0ecf73662359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:22.770150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.770101    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c84a28-d678-4a81-a8f5-0ecf73662359-kube-api-access-hl5wf" pod "b9c84a28-d678-4a81-a8f5-0ecf73662359" (UID: "b9c84a28-d678-4a81-a8f5-0ecf73662359"). InnerVolumeSpecName "kube-api-access-hl5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:22.771425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.771400    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-dshm" pod "b9c84a28-d678-4a81-a8f5-0ecf73662359" (UID: "b9c84a28-d678-4a81-a8f5-0ecf73662359"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:22.869380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.869340    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c84a28-d678-4a81-a8f5-0ecf73662359-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:22.869380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.869374    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5wf\" (UniqueName: \"kubernetes.io/projected/b9c84a28-d678-4a81-a8f5-0ecf73662359-kube-api-access-hl5wf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:22.869380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.869386    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c84a28-d678-4a81-a8f5-0ecf73662359-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:22.955853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.955809    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"]
Oct 30 01:17:22.956767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956747    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" containerName="main"
Oct 30 01:17:22.956767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956765    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" containerName="main"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956787    5104 cpu_manager.go:414] "containerMap: removing container" podUID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" containerName="payload-processing"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956792    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" containerName="payload-processing"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956818    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956824    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956835    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6d148f0d-2186-472c-81df-0d88cef41a48" containerName="revoke-keys"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956840    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d148f0d-2186-472c-81df-0d88cef41a48" containerName="revoke-keys"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956861    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" containerName="payload-pre-processing"
Oct 30 01:17:22.956935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.956866    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" containerName="payload-pre-processing"
Oct 30 01:17:22.957238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.957001    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d4415e-f0a4-4ce4-bfbb-4b63b590f9aa" containerName="main"
Oct 30 01:17:22.957238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.957011    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f33366-e2a3-4e2b-86a5-4924e65d3b69" containerName="payload-processing"
Oct 30 01:17:22.957238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.957021    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main"
Oct 30 01:17:22.957238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.957027    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d148f0d-2186-472c-81df-0d88cef41a48" containerName="revoke-keys"
Oct 30 01:17:22.957238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.957035    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa431ff4-1718-443e-ad8b-7a60ca7e7a3d" containerName="payload-pre-processing"
Oct 30 01:17:22.961341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.961323    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:22.975560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.975532    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:22.975560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.975543    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:22.980614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.980481    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"]
Oct 30 01:17:22.983607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.983583    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:17:22.983785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:22.983609    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:23.073189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.073089    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.073387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.073372    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.073657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.073627    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpp6b\" (UniqueName: \"kubernetes.io/projected/208f7391-3f08-434e-aa58-8652e6e34ac0-kube-api-access-qpp6b\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.175189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.175142    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpp6b\" (UniqueName: \"kubernetes.io/projected/208f7391-3f08-434e-aa58-8652e6e34ac0-kube-api-access-qpp6b\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.175400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.175239    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.175549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.175530    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.176100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.176073    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.178085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.178062    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.199623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.199592    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpp6b\" (UniqueName: \"kubernetes.io/projected/208f7391-3f08-434e-aa58-8652e6e34ac0-kube-api-access-qpp6b\") pod \"maas-api-revoke-sub-bbba2121-zhqhn\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.277929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.277889    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:23.453017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.452959    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.232:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:23.546999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.546971    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"
Oct 30 01:17:23.546999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.546991    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk" event={"ID":"b9c84a28-d678-4a81-a8f5-0ecf73662359","Type":"ContainerDied","Data":"c954f31e0a4e5071f092702b280e7d0473bc4b71fd57d36ae8221bb6866c3d78"}
Oct 30 01:17:23.547226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.547041    5104 scope.go:122] "RemoveContainer" containerID="3c66a64312fe5aac1754bcb9db178c87006678a90cb6ec1324e057136d0509b1"
Oct 30 01:17:23.613644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.613596    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"]
Oct 30 01:17:23.626615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.626570    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"]
Oct 30 01:17:23.634712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:23.634683    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/auth-test-model-eacea3-kserve-669498dd5-rxqjk"]
Oct 30 01:17:24.149662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.149564    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"]
Oct 30 01:17:24.149836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.149773    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" containerID="cri-o://ddcd629386378411440037556c0d7d7859804e37a7c916b6a10efbfff7ce1d4a" gracePeriod=30
Oct 30 01:17:24.291063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.291023    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"]
Oct 30 01:17:24.291655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.291289    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy" containerID="cri-o://70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea" gracePeriod=30
Oct 30 01:17:24.610995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.610957    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c84a28-d678-4a81-a8f5-0ecf73662359" path="/var/lib/kubelet/pods/b9c84a28-d678-4a81-a8f5-0ecf73662359/volumes"
Oct 30 01:17:24.611542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.611509    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn" event={"ID":"208f7391-3f08-434e-aa58-8652e6e34ac0","Type":"ContainerStarted","Data":"25ff49f286f26f2d63a0ec1c467ac563522255f807ab81770bcaf99d6d6054fe"}
Oct 30 01:17:24.611645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.611567    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn" event={"ID":"208f7391-3f08-434e-aa58-8652e6e34ac0","Type":"ContainerStarted","Data":"0cad435adceb92b23336c0ee10611c07aa1dda07af884cba18152c5dfc118e9e"}
Oct 30 01:17:24.649432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:24.649369    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn" podStartSLOduration=2.649351646 podStartE2EDuration="2.649351646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:24.644314997 +0000 UTC m=+2086.192005492" watchObservedRunningTime="2026-10-30 01:17:24.649351646 +0000 UTC m=+2086.197042145"
Oct 30 01:17:25.333623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.333579    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" podUID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" containerName="main" containerID="cri-o://114ef00aa6842c8ce6cb71fbfb122236959c4fd1400682a7b31242e5220699f2" gracePeriod=45
Oct 30 01:17:25.365355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.365310    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" podUID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.239:8000/ready\": dial tcp 10.132.0.239:8000: connect: connection refused"
Oct 30 01:17:25.621749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.621657    5104 generic.go:356] "Generic (PLEG): container finished" podID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" containerID="114ef00aa6842c8ce6cb71fbfb122236959c4fd1400682a7b31242e5220699f2" exitCode=0
Oct 30 01:17:25.621749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.621726    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" event={"ID":"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10","Type":"ContainerDied","Data":"114ef00aa6842c8ce6cb71fbfb122236959c4fd1400682a7b31242e5220699f2"}
Oct 30 01:17:25.624095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.624064    5104 generic.go:356] "Generic (PLEG): container finished" podID="208f7391-3f08-434e-aa58-8652e6e34ac0" containerID="25ff49f286f26f2d63a0ec1c467ac563522255f807ab81770bcaf99d6d6054fe" exitCode=0
Oct 30 01:17:25.624237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.624145    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn" event={"ID":"208f7391-3f08-434e-aa58-8652e6e34ac0","Type":"ContainerDied","Data":"25ff49f286f26f2d63a0ec1c467ac563522255f807ab81770bcaf99d6d6054fe"}
Oct 30 01:17:25.692922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.692888    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:17:25.860272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860237    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-model-cache\") pod \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") "
Oct 30 01:17:25.860428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860347    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tmp-dir\") pod \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") "
Oct 30 01:17:25.860428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860373    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tls-certs\" (UniqueName: \"kubernetes.io/secret/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tls-certs\") pod \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") "
Oct 30 01:17:25.860560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860449    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-kube-api-access-rlzlv\" (UniqueName: \"kubernetes.io/projected/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-kube-api-access-rlzlv\") pod \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") "
Oct 30 01:17:25.860560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860494    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-model-cache" pod "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" (UID: "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:25.860560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860539    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-home\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-home\") pod \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") "
Oct 30 01:17:25.860696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860584    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-dshm\") pod \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\" (UID: \"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10\") "
Oct 30 01:17:25.860696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860660    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tmp-dir" pod "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" (UID: "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:25.860840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.860819    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-home" pod "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" (UID: "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:25.861713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.861676    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:25.861713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.861708    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:25.861904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.861724    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:25.862978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.862949    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-dshm" pod "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" (UID: "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:25.863081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.863049    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-kube-api-access-rlzlv" pod "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" (UID: "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10"). InnerVolumeSpecName "kube-api-access-rlzlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:25.863206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.863181    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tls-certs" pod "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" (UID: "44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:25.963162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.963129    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:25.963162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.963161    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzlv\" (UniqueName: \"kubernetes.io/projected/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-kube-api-access-rlzlv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:25.963429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:25.963172    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:26.635539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.635509    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"
Oct 30 01:17:26.635539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.635529    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v" event={"ID":"44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10","Type":"ContainerDied","Data":"21b6e51b51ca49075ffd7b8e3d8bd2f4cfd3308760800eb316f27a43e9c0fca2"}
Oct 30 01:17:26.636084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.635579    5104 scope.go:122] "RemoveContainer" containerID="114ef00aa6842c8ce6cb71fbfb122236959c4fd1400682a7b31242e5220699f2"
Oct 30 01:17:26.692278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.691183    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"]
Oct 30 01:17:26.702616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.702588    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/auth-test-model-69dc90-kserve-7c488c7557-vh79v"]
Oct 30 01:17:26.894489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.879172    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:26.943480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.935066    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208f7391-3f08-434e-aa58-8652e6e34ac0-kube-api-access-qpp6b\" (UniqueName: \"kubernetes.io/projected/208f7391-3f08-434e-aa58-8652e6e34ac0-kube-api-access-qpp6b\") pod \"208f7391-3f08-434e-aa58-8652e6e34ac0\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") "
Oct 30 01:17:26.943480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.935133    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-xks-ca\") pod \"208f7391-3f08-434e-aa58-8652e6e34ac0\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") "
Oct 30 01:17:26.943480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.935349    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-openshift-ca\") pod \"208f7391-3f08-434e-aa58-8652e6e34ac0\" (UID: \"208f7391-3f08-434e-aa58-8652e6e34ac0\") "
Oct 30 01:17:26.971485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:26.955026    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-openshift-ca" pod "208f7391-3f08-434e-aa58-8652e6e34ac0" (UID: "208f7391-3f08-434e-aa58-8652e6e34ac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:27.014678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.014597    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-xks-ca" pod "208f7391-3f08-434e-aa58-8652e6e34ac0" (UID: "208f7391-3f08-434e-aa58-8652e6e34ac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:27.015050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.015020    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208f7391-3f08-434e-aa58-8652e6e34ac0-kube-api-access-qpp6b" pod "208f7391-3f08-434e-aa58-8652e6e34ac0" (UID: "208f7391-3f08-434e-aa58-8652e6e34ac0"). InnerVolumeSpecName "kube-api-access-qpp6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:27.072489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.052240    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.072489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.052285    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpp6b\" (UniqueName: \"kubernetes.io/projected/208f7391-3f08-434e-aa58-8652e6e34ac0-kube-api-access-qpp6b\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.072489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.052304    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208f7391-3f08-434e-aa58-8652e6e34ac0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.369880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.369843    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"]
Oct 30 01:17:27.370857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.370838    5104 cpu_manager.go:414] "containerMap: removing container" podUID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" containerName="main"
Oct 30 01:17:27.370943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.370858    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" containerName="main"
Oct 30 01:17:27.370943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.370907    5104 cpu_manager.go:414] "containerMap: removing container" podUID="208f7391-3f08-434e-aa58-8652e6e34ac0" containerName="revoke-keys"
Oct 30 01:17:27.370943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.370912    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208f7391-3f08-434e-aa58-8652e6e34ac0" containerName="revoke-keys"
Oct 30 01:17:27.371058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.371048    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" containerName="main"
Oct 30 01:17:27.371103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.371068    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="208f7391-3f08-434e-aa58-8652e6e34ac0" containerName="revoke-keys"
Oct 30 01:17:27.375403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.375372    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.395742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.395708    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"]
Oct 30 01:17:27.459193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.459154    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.459405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.459295    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwslh\" (UniqueName: \"kubernetes.io/projected/8cab7767-f495-41db-91c7-2542fb0af7cf-kube-api-access-pwslh\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.459655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.459608    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.561537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.561502    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.561702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.561569    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwslh\" (UniqueName: \"kubernetes.io/projected/8cab7767-f495-41db-91c7-2542fb0af7cf-kube-api-access-pwslh\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.561792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.561708    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.562348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.562304    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.564326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.564298    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.581736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.581701    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwslh\" (UniqueName: \"kubernetes.io/projected/8cab7767-f495-41db-91c7-2542fb0af7cf-kube-api-access-pwslh\") pod \"maas-api-revoke-sub-1fb342b8-vdmvx\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.704186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.698228    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:27.768807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.768779    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"
Oct 30 01:17:27.769132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.769060    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn" event={"ID":"208f7391-3f08-434e-aa58-8652e6e34ac0","Type":"ContainerDied","Data":"0cad435adceb92b23336c0ee10611c07aa1dda07af884cba18152c5dfc118e9e"}
Oct 30 01:17:27.769132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:27.769112    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cad435adceb92b23336c0ee10611c07aa1dda07af884cba18152c5dfc118e9e"
Oct 30 01:17:28.070014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:28.069983    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"]
Oct 30 01:17:28.070187 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:17:28.070164    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cab7767_f495_41db_91c7_2542fb0af7cf.slice/crio-ee29ada6a9add9bdcdad274a8951dd1197ae35160bf2229c7e7972e154a9e930 WatchSource:0}: Error finding container ee29ada6a9add9bdcdad274a8951dd1197ae35160bf2229c7e7972e154a9e930: Status 404 returned error can't find the container with id ee29ada6a9add9bdcdad274a8951dd1197ae35160bf2229c7e7972e154a9e930
Oct 30 01:17:28.583394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:28.583350    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10" path="/var/lib/kubelet/pods/44ad83d4-e9ba-43a8-9d9a-c69a94c7ad10/volumes"
Oct 30 01:17:28.682367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:28.682337    5104 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:17:28.682728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:28.682689    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:17:28.835650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:28.832896    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx" event={"ID":"8cab7767-f495-41db-91c7-2542fb0af7cf","Type":"ContainerStarted","Data":"68dc0499127da8365dd540053111828fea9d30260ba38c134df06556916da0c9"}
Oct 30 01:17:28.835650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:28.832969    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx" event={"ID":"8cab7767-f495-41db-91c7-2542fb0af7cf","Type":"ContainerStarted","Data":"ee29ada6a9add9bdcdad274a8951dd1197ae35160bf2229c7e7972e154a9e930"}
Oct 30 01:17:29.804531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:29.763741    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 01:17:29.804531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:29.763804    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 01:17:30.103538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.102053    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" podUID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.238:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:30.168631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.106722    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:30.168883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.148891    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx" podStartSLOduration=3.148868763 podStartE2EDuration="3.148868763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:28.909780403 +0000 UTC m=+2090.457470889" watchObservedRunningTime="2026-10-30 01:17:30.148868763 +0000 UTC m=+2091.696559259"
Oct 30 01:17:30.198480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.197689    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"]
Oct 30 01:17:30.222491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.207240    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52f1cb-8bp4c"]
Oct 30 01:17:30.306188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.272223    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5" podUID="b5802f4e-a81e-4630-98ce-b308b35f457b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.242:8000/ready\": context deadline exceeded"
Oct 30 01:17:30.306188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.277794    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:30.323962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.322999    5104 generic.go:356] "Generic (PLEG): container finished" podID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerID="ddcd629386378411440037556c0d7d7859804e37a7c916b6a10efbfff7ce1d4a" exitCode=0
Oct 30 01:17:30.329566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.324980    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" event={"ID":"3c26b785-d2fc-428b-b7cd-14dd05db4270","Type":"ContainerDied","Data":"ddcd629386378411440037556c0d7d7859804e37a7c916b6a10efbfff7ce1d4a"}
Oct 30 01:17:30.411396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.390616    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg" podUID="e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" containerName="main" probeResult="failure" output="Get \"https://10.132.1.6:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:30.485479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.481549    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:17:30.576002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.575958    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:30.592732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.592682    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:30.615783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:30.615745    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cbe8731-8b97-4e3d-a673-85b5abe7df28" path="/var/lib/kubelet/pods/1cbe8731-8b97-4e3d-a673-85b5abe7df28/volumes"
Oct 30 01:17:31.194995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.194965    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.389608    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-data\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.389777    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-envoy\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.389822    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-socket\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.389875    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istiod-ca-cert\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.389965    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-credential-socket\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.390028    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-podinfo\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.390133    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-kube-api-access-hv7nn\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-kube-api-access-hv7nn\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.390280    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-certs\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.391315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.390352    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-token\") pod \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\" (UID: \"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8\") "
Oct 30 01:17:31.394600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.394270    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-data" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.411440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.409780    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-certs" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.411440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.411078    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-socket" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.413441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.413124    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-credential-socket" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.413441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.413286    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istiod-ca-cert" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:31.465305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.463232    5104 generic.go:356] "Generic (PLEG): container finished" podID="8cab7767-f495-41db-91c7-2542fb0af7cf" containerID="68dc0499127da8365dd540053111828fea9d30260ba38c134df06556916da0c9" exitCode=0
Oct 30 01:17:31.465305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.463582    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx" event={"ID":"8cab7767-f495-41db-91c7-2542fb0af7cf","Type":"ContainerDied","Data":"68dc0499127da8365dd540053111828fea9d30260ba38c134df06556916da0c9"}
Oct 30 01:17:31.488136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.488091    5104 generic.go:356] "Generic (PLEG): container finished" podID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerID="70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea" exitCode=0
Oct 30 01:17:31.488327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.488266    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" event={"ID":"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8","Type":"ContainerDied","Data":"70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea"}
Oct 30 01:17:31.488327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.488321    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48" event={"ID":"c97970c7-2b74-4a0d-b5f6-698ebbe69bd8","Type":"ContainerDied","Data":"08546feb24d59a6ae8928007b57abe0ecc5d7dafba64494ed0c6585c91d1717c"}
Oct 30 01:17:31.488491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.488352    5104 scope.go:122] "RemoveContainer" containerID="70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea"
Oct 30 01:17:31.488578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.488563    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"
Oct 30 01:17:31.517271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.516852    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.517271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.516893    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.517271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.516908    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.517271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.516923    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.517271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.516939    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.520753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.520717    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-podinfo" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:31.520986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.520962    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-envoy" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.523685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.523652    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-kube-api-access-hv7nn" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "kube-api-access-hv7nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:31.524552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.524525    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-token" pod "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" (UID: "c97970c7-2b74-4a0d-b5f6-698ebbe69bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:31.537814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.537782    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 01:17:31.543452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.543389    5104 scope.go:122] "RemoveContainer" containerID="70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea"
Oct 30 01:17:31.543964 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:31.543932    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea\": container with ID starting with 70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea not found: ID does not exist" containerID="70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea"
Oct 30 01:17:31.544047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.543978    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea"} err="failed to get container status \"70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea\": rpc error: code = NotFound desc = could not find container \"70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea\": container with ID starting with 70b37d230ff39f9eefdde26659d0b9b86429249855634f60df7df3d59357aeea not found: ID does not exist"
Oct 30 01:17:31.617880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.617841    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-envoy\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.618077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.617948    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-socket\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.618077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.617982    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-data\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.618201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618106    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-token\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-token\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.618201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618161    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c26b785-d2fc-428b-b7cd-14dd05db4270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c26b785-d2fc-428b-b7cd-14dd05db4270-istiod-ca-cert\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.618369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618349    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-credential-socket\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.618449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618359    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-socket" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.618595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618572    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-kube-api-access-fjqwv\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-kube-api-access-fjqwv\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.618866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618583    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-data" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.618866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618751    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-credential-socket" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.618866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.618821    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c26b785-d2fc-428b-b7cd-14dd05db4270-istiod-ca-cert" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:31.619130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.619105    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-podinfo\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.619243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.619168    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-certs\") pod \"3c26b785-d2fc-428b-b7cd-14dd05db4270\" (UID: \"3c26b785-d2fc-428b-b7cd-14dd05db4270\") "
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.619851    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-certs" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620362    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620388    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620404    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620417    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c26b785-d2fc-428b-b7cd-14dd05db4270-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620432    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7nn\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-kube-api-access-hv7nn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620445    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620477    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620493    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.620526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.620505    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.621183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.621156    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-token" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:31.621734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.621700    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-kube-api-access-fjqwv" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "kube-api-access-fjqwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:31.621832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.621765    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-podinfo" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:31.621832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.621781    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-envoy" pod "3c26b785-d2fc-428b-b7cd-14dd05db4270" (UID: "3c26b785-d2fc-428b-b7cd-14dd05db4270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:31.730519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.730442    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqwv\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-kube-api-access-fjqwv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.730519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.730496    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.730519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.730512    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.730519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.730524    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c26b785-d2fc-428b-b7cd-14dd05db4270-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:31.863731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.863672    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"]
Oct 30 01:17:31.873618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:31.873572    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-ghs48"]
Oct 30 01:17:32.159132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.159092    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5" podUID="0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" containerName="main" containerID="cri-o://6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56" gracePeriod=45
Oct 30 01:17:32.479249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.479219    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:17:32.501036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.500893    5104 generic.go:356] "Generic (PLEG): container finished" podID="0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" containerID="6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56" exitCode=0
Oct 30 01:17:32.501036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.500949    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"
Oct 30 01:17:32.501036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.500948    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5" event={"ID":"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4","Type":"ContainerDied","Data":"6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56"}
Oct 30 01:17:32.501375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.501067    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5" event={"ID":"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4","Type":"ContainerDied","Data":"c6fb1ef5a8e5b731c1a5b8423de4725d1716a3c72dbbb0b712dfa37cfac726c3"}
Oct 30 01:17:32.501375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.501093    5104 scope.go:122] "RemoveContainer" containerID="6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56"
Oct 30 01:17:32.504716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.504691    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"
Oct 30 01:17:32.504833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.504714    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq" event={"ID":"3c26b785-d2fc-428b-b7cd-14dd05db4270","Type":"ContainerDied","Data":"3d286636fa934f55c6b0f4f9f56d223ceaf18d0b601ac59f4488334a6bf675ca"}
Oct 30 01:17:32.515952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.515912    5104 scope.go:122] "RemoveContainer" containerID="6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56"
Oct 30 01:17:32.516250 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:32.516228    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56\": container with ID starting with 6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56 not found: ID does not exist" containerID="6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56"
Oct 30 01:17:32.516333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.516255    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56"} err="failed to get container status \"6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56\": rpc error: code = NotFound desc = could not find container \"6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56\": container with ID starting with 6a906013f25748158153ebac80007cf6854650a1692600fd98efff0511ce7e56 not found: ID does not exist"
Oct 30 01:17:32.516333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.516275    5104 scope.go:122] "RemoveContainer" containerID="ddcd629386378411440037556c0d7d7859804e37a7c916b6a10efbfff7ce1d4a"
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.551431    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-dshm\") pod \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") "
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.551590    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tls-certs\") pod \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") "
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.551661    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-model-cache\") pod \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") "
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.551737    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-home\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-home\") pod \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") "
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.552055    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-model-cache" pod "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" (UID: "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.552315    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-home" pod "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" (UID: "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.552973    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-kube-api-access-rxdlc\") pod \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") "
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.553536    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tmp-dir\") pod \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\" (UID: \"0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4\") "
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.553812    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tmp-dir" pod "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" (UID: "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.554680    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.554703    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.555375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.554718    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.556216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.555640    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-kube-api-access-rxdlc" pod "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" (UID: "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4"). InnerVolumeSpecName "kube-api-access-rxdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:32.559430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.559399    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tls-certs" pod "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" (UID: "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:32.567231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.564325    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-dshm" pod "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" (UID: "0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:32.575278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.568497    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"]
Oct 30 01:17:32.594304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.594194    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" path="/var/lib/kubelet/pods/c97970c7-2b74-4a0d-b5f6-698ebbe69bd8/volumes"
Oct 30 01:17:32.596651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.596439    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-gpvwq"]
Oct 30 01:17:32.666348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.666309    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-kube-api-access-rxdlc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.666348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.666350    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.666662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.666367    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.724501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.724472    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:32.842429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.842370    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"]
Oct 30 01:17:32.848006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.847932    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/auth-test-model-29145c-kserve-5986d669f7-mwkx5"]
Oct 30 01:17:32.871562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.871524    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cab7767-f495-41db-91c7-2542fb0af7cf-kube-api-access-pwslh\" (UniqueName: \"kubernetes.io/projected/8cab7767-f495-41db-91c7-2542fb0af7cf-kube-api-access-pwslh\") pod \"8cab7767-f495-41db-91c7-2542fb0af7cf\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") "
Oct 30 01:17:32.871736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.871721    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-openshift-ca\") pod \"8cab7767-f495-41db-91c7-2542fb0af7cf\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") "
Oct 30 01:17:32.871866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.871849    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-xks-ca\") pod \"8cab7767-f495-41db-91c7-2542fb0af7cf\" (UID: \"8cab7767-f495-41db-91c7-2542fb0af7cf\") "
Oct 30 01:17:32.872146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.872106    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-openshift-ca" pod "8cab7767-f495-41db-91c7-2542fb0af7cf" (UID: "8cab7767-f495-41db-91c7-2542fb0af7cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:32.872644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.872617    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.874057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.874029    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cab7767-f495-41db-91c7-2542fb0af7cf-kube-api-access-pwslh" pod "8cab7767-f495-41db-91c7-2542fb0af7cf" (UID: "8cab7767-f495-41db-91c7-2542fb0af7cf"). InnerVolumeSpecName "kube-api-access-pwslh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:32.874215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.874137    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-xks-ca" pod "8cab7767-f495-41db-91c7-2542fb0af7cf" (UID: "8cab7767-f495-41db-91c7-2542fb0af7cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:32.974335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.974298    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cab7767-f495-41db-91c7-2542fb0af7cf-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:32.974335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:32.974334    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwslh\" (UniqueName: \"kubernetes.io/projected/8cab7767-f495-41db-91c7-2542fb0af7cf-kube-api-access-pwslh\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:33.518496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:33.518425    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"
Oct 30 01:17:33.518496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:33.518466    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx" event={"ID":"8cab7767-f495-41db-91c7-2542fb0af7cf","Type":"ContainerDied","Data":"ee29ada6a9add9bdcdad274a8951dd1197ae35160bf2229c7e7972e154a9e930"}
Oct 30 01:17:33.519058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:33.518527    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee29ada6a9add9bdcdad274a8951dd1197ae35160bf2229c7e7972e154a9e930"
Oct 30 01:17:34.584126 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:34.584082    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" path="/var/lib/kubelet/pods/0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4/volumes"
Oct 30 01:17:34.584874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:34.584855    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" path="/var/lib/kubelet/pods/3c26b785-d2fc-428b-b7cd-14dd05db4270/volumes"
Oct 30 01:17:36.926591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:36.926552    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"]
Oct 30 01:17:36.927222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:36.926871    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy" containerID="cri-o://8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60" gracePeriod=30
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.446159    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"]
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.447881    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.447906    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.447931    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.447939    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.447952    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8cab7767-f495-41db-91c7-2542fb0af7cf" containerName="revoke-keys"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.447960    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cab7767-f495-41db-91c7-2542fb0af7cf" containerName="revoke-keys"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.447999    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" containerName="main"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.448007    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" containerName="main"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.448281    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97970c7-2b74-4a0d-b5f6-698ebbe69bd8" containerName="istio-proxy"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.448301    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c26b785-d2fc-428b-b7cd-14dd05db4270" containerName="istio-proxy"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.448311    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cab7767-f495-41db-91c7-2542fb0af7cf" containerName="revoke-keys"
Oct 30 01:17:37.451482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.448327    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0880f2a7-43c9-4e7d-9a46-50e9e8d7a5c4" containerName="main"
Oct 30 01:17:37.456487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.455014    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.478532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.476907    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"]
Oct 30 01:17:37.492489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.490794    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:17:37.492489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.491500    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:37.492489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.491591    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:37.492489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.491637    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:37.539570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.538495    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.539570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.538603    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.539570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.538671    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkkz\" (UniqueName: \"kubernetes.io/projected/28e758f3-e8be-472b-8b20-a69d2f26794e-kube-api-access-xdkkz\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.640071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.640026    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkkz\" (UniqueName: \"kubernetes.io/projected/28e758f3-e8be-472b-8b20-a69d2f26794e-kube-api-access-xdkkz\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.640487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.640447    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.640607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.640595    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.641317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.641282    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.642963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.642938    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.660199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.660169    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkkz\" (UniqueName: \"kubernetes.io/projected/28e758f3-e8be-472b-8b20-a69d2f26794e-kube-api-access-xdkkz\") pod \"maas-api-revoke-keys-e2e-inf-b-687471-q2s8z\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:37.860734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:37.860644    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:38.142115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.142033    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"]
Oct 30 01:17:38.182765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.182693    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"]
Oct 30 01:17:38.183395 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:17:38.183361    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e758f3_e8be_472b_8b20_a69d2f26794e.slice/crio-4af4be580033114329fff5ade1b7bb0648f1ac2f27fc1734fbbf752e1f4a4290 WatchSource:0}: Error finding container 4af4be580033114329fff5ade1b7bb0648f1ac2f27fc1734fbbf752e1f4a4290: Status 404 returned error can't find the container with id 4af4be580033114329fff5ade1b7bb0648f1ac2f27fc1734fbbf752e1f4a4290
Oct 30 01:17:38.352495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.346530    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"]
Oct 30 01:17:38.380654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.379405    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.424977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.423659    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"]
Oct 30 01:17:38.467524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.467484    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.467727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.467618    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.467727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.467669    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcdd\" (UniqueName: \"kubernetes.io/projected/635f89ba-f80b-4b1f-8205-74a867b78ebf-kube-api-access-mkcdd\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.569575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.569538    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.569827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.569599    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.569827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.569634    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcdd\" (UniqueName: \"kubernetes.io/projected/635f89ba-f80b-4b1f-8205-74a867b78ebf-kube-api-access-mkcdd\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.570593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.570548    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.576096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.576062    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.590655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.590619    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcdd\" (UniqueName: \"kubernetes.io/projected/635f89ba-f80b-4b1f-8205-74a867b78ebf-kube-api-access-mkcdd\") pod \"maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:38.592156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.592120    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z" event={"ID":"28e758f3-e8be-472b-8b20-a69d2f26794e","Type":"ContainerStarted","Data":"4af4be580033114329fff5ade1b7bb0648f1ac2f27fc1734fbbf752e1f4a4290"}
Oct 30 01:17:38.724399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:38.724362    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:39.112346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.112315    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:17:39.112893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.112873    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:17:39.122182 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:39.122155    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:17:39.124371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.124352    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:17:39.126180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.126161    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:17:39.366174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.366081    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"]
Oct 30 01:17:39.608943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.608895    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z" event={"ID":"28e758f3-e8be-472b-8b20-a69d2f26794e","Type":"ContainerStarted","Data":"f0d293708146bbd81a223903d984b9498f566e794165ebd038e54b7d9f2c928b"}
Oct 30 01:17:39.611285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.611253    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs" event={"ID":"635f89ba-f80b-4b1f-8205-74a867b78ebf","Type":"ContainerStarted","Data":"ce7cc26574dd18350f02ad98f5d930606f35b39e679b7edd33570c33ee8718db"}
Oct 30 01:17:39.648419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:39.648310    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z" podStartSLOduration=2.648295355 podStartE2EDuration="2.648295355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:39.646918925 +0000 UTC m=+2101.194609422" watchObservedRunningTime="2026-10-30 01:17:39.648295355 +0000 UTC m=+2101.195985868"
Oct 30 01:17:40.623791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:40.623748    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs" event={"ID":"635f89ba-f80b-4b1f-8205-74a867b78ebf","Type":"ContainerStarted","Data":"033b600d4763c483b96281b8a823f128fbfcc4cb2469f6f02375c15f3d81369e"}
Oct 30 01:17:40.625709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:40.625678    5104 generic.go:356] "Generic (PLEG): container finished" podID="28e758f3-e8be-472b-8b20-a69d2f26794e" containerID="f0d293708146bbd81a223903d984b9498f566e794165ebd038e54b7d9f2c928b" exitCode=0
Oct 30 01:17:40.625865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:40.625770    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z" event={"ID":"28e758f3-e8be-472b-8b20-a69d2f26794e","Type":"ContainerDied","Data":"f0d293708146bbd81a223903d984b9498f566e794165ebd038e54b7d9f2c928b"}
Oct 30 01:17:40.713536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:40.713429    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs" podStartSLOduration=2.713409071 podStartE2EDuration="2.713409071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:40.669910315 +0000 UTC m=+2102.217600811" watchObservedRunningTime="2026-10-30 01:17:40.713409071 +0000 UTC m=+2102.261099567"
Oct 30 01:17:41.637528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.637492    5104 generic.go:356] "Generic (PLEG): container finished" podID="635f89ba-f80b-4b1f-8205-74a867b78ebf" containerID="033b600d4763c483b96281b8a823f128fbfcc4cb2469f6f02375c15f3d81369e" exitCode=0
Oct 30 01:17:41.638023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.637528    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs" event={"ID":"635f89ba-f80b-4b1f-8205-74a867b78ebf","Type":"ContainerDied","Data":"033b600d4763c483b96281b8a823f128fbfcc4cb2469f6f02375c15f3d81369e"}
Oct 30 01:17:41.888350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.888269    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:41.962953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.962919    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-openshift-ca\") pod \"28e758f3-e8be-472b-8b20-a69d2f26794e\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") "
Oct 30 01:17:41.963145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.963073    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e758f3-e8be-472b-8b20-a69d2f26794e-kube-api-access-xdkkz\" (UniqueName: \"kubernetes.io/projected/28e758f3-e8be-472b-8b20-a69d2f26794e-kube-api-access-xdkkz\") pod \"28e758f3-e8be-472b-8b20-a69d2f26794e\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") "
Oct 30 01:17:41.963219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.963159    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-xks-ca\") pod \"28e758f3-e8be-472b-8b20-a69d2f26794e\" (UID: \"28e758f3-e8be-472b-8b20-a69d2f26794e\") "
Oct 30 01:17:41.963377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.963348    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-openshift-ca" pod "28e758f3-e8be-472b-8b20-a69d2f26794e" (UID: "28e758f3-e8be-472b-8b20-a69d2f26794e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:41.964171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.964141    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:41.965859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.965832    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e758f3-e8be-472b-8b20-a69d2f26794e-kube-api-access-xdkkz" pod "28e758f3-e8be-472b-8b20-a69d2f26794e" (UID: "28e758f3-e8be-472b-8b20-a69d2f26794e"). InnerVolumeSpecName "kube-api-access-xdkkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:41.967577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:41.967555    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-xks-ca" pod "28e758f3-e8be-472b-8b20-a69d2f26794e" (UID: "28e758f3-e8be-472b-8b20-a69d2f26794e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:42.066011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.065962    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdkkz\" (UniqueName: \"kubernetes.io/projected/28e758f3-e8be-472b-8b20-a69d2f26794e-kube-api-access-xdkkz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.066011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.066007    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e758f3-e8be-472b-8b20-a69d2f26794e-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.251066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.251039    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:17:42.370753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.370698    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-socket\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.370753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.370741    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-data\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.370809    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-envoy\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.370851    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-podinfo\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.370901    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istiod-ca-cert\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.371010    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-socket" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:42.371265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.371052    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-token\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-token\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.371198    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-certs\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.371240    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-kube-api-access-zz5tc\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-kube-api-access-zz5tc\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.371293    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-credential-socket\") pod \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\" (UID: \"3fc8515d-c0ef-4f1f-9c23-536f9340fb74\") "
Oct 30 01:17:42.371951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.371925    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-data" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:42.372530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.372431    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.372530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.372480    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.373015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.372521    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-credential-socket" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:42.374371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.374345    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istiod-ca-cert" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:42.374617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.374588    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-envoy" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:42.375930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.375892    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-token" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:42.377244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.377215    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-podinfo" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:42.377334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.377309    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-kube-api-access-zz5tc" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "kube-api-access-zz5tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:42.377579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.377560    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-certs" pod "3fc8515d-c0ef-4f1f-9c23-536f9340fb74" (UID: "3fc8515d-c0ef-4f1f-9c23-536f9340fb74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:42.485537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.475161    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.485537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.475201    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.485537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.475248    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz5tc\" (UniqueName: \"kubernetes.io/projected/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-kube-api-access-zz5tc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.485537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.475265    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.485537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.475294    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.485537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.475308    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.485537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.475320    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc8515d-c0ef-4f1f-9c23-536f9340fb74-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.649957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.649925    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"
Oct 30 01:17:42.650380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.649950    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z" event={"ID":"28e758f3-e8be-472b-8b20-a69d2f26794e","Type":"ContainerDied","Data":"4af4be580033114329fff5ade1b7bb0648f1ac2f27fc1734fbbf752e1f4a4290"}
Oct 30 01:17:42.650380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.650005    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af4be580033114329fff5ade1b7bb0648f1ac2f27fc1734fbbf752e1f4a4290"
Oct 30 01:17:42.651725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.651689    5104 generic.go:356] "Generic (PLEG): container finished" podID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerID="8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60" exitCode=0
Oct 30 01:17:42.651890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.651768    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"
Oct 30 01:17:42.651890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.651772    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" event={"ID":"3fc8515d-c0ef-4f1f-9c23-536f9340fb74","Type":"ContainerDied","Data":"8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60"}
Oct 30 01:17:42.651890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.651828    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh" event={"ID":"3fc8515d-c0ef-4f1f-9c23-536f9340fb74","Type":"ContainerDied","Data":"099aa262187982a58b55e89c07647fd2a95ce88d0e329ac9dd7246cb4bc1ec2c"}
Oct 30 01:17:42.651890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.651857    5104 scope.go:122] "RemoveContainer" containerID="8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60"
Oct 30 01:17:42.672618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.672591    5104 scope.go:122] "RemoveContainer" containerID="8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60"
Oct 30 01:17:42.673070 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:42.673038    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60\": container with ID starting with 8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60 not found: ID does not exist" containerID="8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60"
Oct 30 01:17:42.673202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.673081    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60"} err="failed to get container status \"8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60\": rpc error: code = NotFound desc = could not find container \"8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60\": container with ID starting with 8b4f8526bbc38eda7d177e9ecc8cfcc20ae42f5e309c3bb1a282e01ba64fed60 not found: ID does not exist"
Oct 30 01:17:42.743553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.734865    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"]
Oct 30 01:17:42.743553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.741771    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-2nhqh"]
Oct 30 01:17:42.826914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.826882    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:42.884275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.884238    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-openshift-ca\") pod \"635f89ba-f80b-4b1f-8205-74a867b78ebf\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") "
Oct 30 01:17:42.884487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.884417    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635f89ba-f80b-4b1f-8205-74a867b78ebf-kube-api-access-mkcdd\" (UniqueName: \"kubernetes.io/projected/635f89ba-f80b-4b1f-8205-74a867b78ebf-kube-api-access-mkcdd\") pod \"635f89ba-f80b-4b1f-8205-74a867b78ebf\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") "
Oct 30 01:17:42.884565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.884552    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-xks-ca\") pod \"635f89ba-f80b-4b1f-8205-74a867b78ebf\" (UID: \"635f89ba-f80b-4b1f-8205-74a867b78ebf\") "
Oct 30 01:17:42.887948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.887916    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-xks-ca" pod "635f89ba-f80b-4b1f-8205-74a867b78ebf" (UID: "635f89ba-f80b-4b1f-8205-74a867b78ebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:42.888468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.888428    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-openshift-ca" pod "635f89ba-f80b-4b1f-8205-74a867b78ebf" (UID: "635f89ba-f80b-4b1f-8205-74a867b78ebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:42.891069 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.891042    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635f89ba-f80b-4b1f-8205-74a867b78ebf-kube-api-access-mkcdd" pod "635f89ba-f80b-4b1f-8205-74a867b78ebf" (UID: "635f89ba-f80b-4b1f-8205-74a867b78ebf"). InnerVolumeSpecName "kube-api-access-mkcdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:42.986574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.986539    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.986574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.986574    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635f89ba-f80b-4b1f-8205-74a867b78ebf-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:42.986806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:42.986588    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkcdd\" (UniqueName: \"kubernetes.io/projected/635f89ba-f80b-4b1f-8205-74a867b78ebf-kube-api-access-mkcdd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:43.662970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:43.662939    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"
Oct 30 01:17:43.663471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:43.662974    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs" event={"ID":"635f89ba-f80b-4b1f-8205-74a867b78ebf","Type":"ContainerDied","Data":"ce7cc26574dd18350f02ad98f5d930606f35b39e679b7edd33570c33ee8718db"}
Oct 30 01:17:43.663471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:43.663033    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7cc26574dd18350f02ad98f5d930606f35b39e679b7edd33570c33ee8718db"
Oct 30 01:17:44.586133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:44.586096    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" path="/var/lib/kubelet/pods/3fc8515d-c0ef-4f1f-9c23-536f9340fb74/volumes"
Oct 30 01:17:46.763178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:46.763144    5104 scope.go:122] "RemoveContainer" containerID="0364c4c248f069dc80326066bf02387d52e89e14334fc6cfaff8b4cef9243c3c"
Oct 30 01:17:46.775441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:46.775417    5104 scope.go:122] "RemoveContainer" containerID="291b8e610ce234f0e0c423457378971133062ae068d8b2e306665baa97d1228f"
Oct 30 01:17:46.787623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:46.787596    5104 scope.go:122] "RemoveContainer" containerID="1c66316f3b436e4a41d8c6007e2017cb22a20a194817e7800f3c2bd6074952b8"
Oct 30 01:17:47.867466 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:47.867421    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"]
Oct 30 01:17:51.476815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.476724    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"]
Oct 30 01:17:51.477871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.477849    5104 cpu_manager.go:414] "containerMap: removing container" podUID="635f89ba-f80b-4b1f-8205-74a867b78ebf" containerName="revoke-keys"
Oct 30 01:17:51.477871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.477873    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635f89ba-f80b-4b1f-8205-74a867b78ebf" containerName="revoke-keys"
Oct 30 01:17:51.477994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.477903    5104 cpu_manager.go:414] "containerMap: removing container" podUID="28e758f3-e8be-472b-8b20-a69d2f26794e" containerName="revoke-keys"
Oct 30 01:17:51.477994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.477908    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e758f3-e8be-472b-8b20-a69d2f26794e" containerName="revoke-keys"
Oct 30 01:17:51.477994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.477935    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy"
Oct 30 01:17:51.477994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.477940    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy"
Oct 30 01:17:51.478128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.478077    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="635f89ba-f80b-4b1f-8205-74a867b78ebf" containerName="revoke-keys"
Oct 30 01:17:51.478128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.478088    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e758f3-e8be-472b-8b20-a69d2f26794e" containerName="revoke-keys"
Oct 30 01:17:51.478128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.478099    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc8515d-c0ef-4f1f-9c23-536f9340fb74" containerName="istio-proxy"
Oct 30 01:17:51.484101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.484082    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.523199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.523161    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"]
Oct 30 01:17:51.544109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.544070    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-29145c-openshift-default-dockercfg-5nhvg\""
Oct 30 01:17:51.552261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.552229    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.552498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.552347    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.552498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.552392    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.552649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.552563    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.552706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.552693    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.552833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.552804    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.553016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.552993    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.553081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.553034    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.553163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.553144    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8t9\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-kube-api-access-hr8t9\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654297    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654346    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654432    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654588    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654656    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654799    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654831    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654857    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.654981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.654937    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8t9\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-kube-api-access-hr8t9\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.655093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.655014    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.655093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.655074    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.655209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.655166    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.655410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.655378    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.655410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.655403    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.656926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.656897    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.657161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.657141    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.674408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.674380    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8t9\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-kube-api-access-hr8t9\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.674531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.674486    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:51.800206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:51.800107    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:52.127675 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:17:52.127644    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62ca0a8_b831_4192_a5e7_2e6d4fe6340c.slice/crio-c5ec2d8e1895da2fddf8fb59842b2277d0b400461d20703f4a9a57f7dd325ff1 WatchSource:0}: Error finding container c5ec2d8e1895da2fddf8fb59842b2277d0b400461d20703f4a9a57f7dd325ff1: Status 404 returned error can't find the container with id c5ec2d8e1895da2fddf8fb59842b2277d0b400461d20703f4a9a57f7dd325ff1
Oct 30 01:17:52.127878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:52.127852    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"]
Oct 30 01:17:52.131312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:52.131275    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:52.131436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:52.131361    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:52.131436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:52.131409    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:17:52.868403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:52.868341    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568" podUID="7803c74d-04f3-46b2-b0e8-552e5ad89b6c" containerName="payload-processing" containerID="cri-o://7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81" gracePeriod=25
Oct 30 01:17:52.933986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:52.933949    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" event={"ID":"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c","Type":"ContainerStarted","Data":"467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2"}
Oct 30 01:17:52.934200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:52.934045    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" event={"ID":"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c","Type":"ContainerStarted","Data":"c5ec2d8e1895da2fddf8fb59842b2277d0b400461d20703f4a9a57f7dd325ff1"}
Oct 30 01:17:53.015862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.015810    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" podStartSLOduration=2.015793359 podStartE2EDuration="2.015793359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:52.994998151 +0000 UTC m=+2114.542688690" watchObservedRunningTime="2026-10-30 01:17:53.015793359 +0000 UTC m=+2114.563483928"
Oct 30 01:17:53.164907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.164872    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" podUID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" containerName="main" containerID="cri-o://fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164" gracePeriod=45
Oct 30 01:17:53.333933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.333905    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:17:53.519888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.519678    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-plugins-config-volume\") pod \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") "
Oct 30 01:17:53.519888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.519748    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-kube-api-access-694rj\" (UniqueName: \"kubernetes.io/projected/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-kube-api-access-694rj\") pod \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\" (UID: \"7803c74d-04f3-46b2-b0e8-552e5ad89b6c\") "
Oct 30 01:17:53.520877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.520847    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-plugins-config-volume" pod "7803c74d-04f3-46b2-b0e8-552e5ad89b6c" (UID: "7803c74d-04f3-46b2-b0e8-552e5ad89b6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:53.523975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.523943    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-kube-api-access-694rj" pod "7803c74d-04f3-46b2-b0e8-552e5ad89b6c" (UID: "7803c74d-04f3-46b2-b0e8-552e5ad89b6c"). InnerVolumeSpecName "kube-api-access-694rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:53.615871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.615481    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:17:53.622358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.622326    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.622358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.622356    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694rj\" (UniqueName: \"kubernetes.io/projected/7803c74d-04f3-46b2-b0e8-552e5ad89b6c-kube-api-access-694rj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.722941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.722898    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-kube-api-access-8pqg8\" (UniqueName: \"kubernetes.io/projected/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-kube-api-access-8pqg8\") pod \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") "
Oct 30 01:17:53.723149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.723043    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-dshm\") pod \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") "
Oct 30 01:17:53.723224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.723182    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-model-cache\") pod \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") "
Oct 30 01:17:53.723335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.723316    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-home\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-home\") pod \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") "
Oct 30 01:17:53.723413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.723399    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tls-certs\") pod \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") "
Oct 30 01:17:53.723488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.723431    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tmp-dir\") pod \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\" (UID: \"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8\") "
Oct 30 01:17:53.724046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.724018    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-home" pod "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" (UID: "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:53.724257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.724236    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-model-cache" pod "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" (UID: "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:53.724510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.724447    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tmp-dir" pod "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" (UID: "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:53.726619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.726541    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.726619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.726565    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.726619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.726577    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.730280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.729251    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-dshm" pod "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" (UID: "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:53.730280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.729730    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tls-certs" pod "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" (UID: "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:53.730280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.730227    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-kube-api-access-8pqg8" pod "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" (UID: "3a747f29-e9f5-455f-9caf-17fb4bb9d1d8"). InnerVolumeSpecName "kube-api-access-8pqg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:53.803859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.803823    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:53.804967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.804942    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 01:17:53.805105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.805011    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 01:17:53.828046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.828010    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.828046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.828046    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.828275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.828062    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pqg8\" (UniqueName: \"kubernetes.io/projected/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8-kube-api-access-8pqg8\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:53.944336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.944244    5104 generic.go:356] "Generic (PLEG): container finished" podID="7803c74d-04f3-46b2-b0e8-552e5ad89b6c" containerID="7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81" exitCode=0
Oct 30 01:17:53.944336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.944292    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"
Oct 30 01:17:53.944336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.944323    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568" event={"ID":"7803c74d-04f3-46b2-b0e8-552e5ad89b6c","Type":"ContainerDied","Data":"7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81"}
Oct 30 01:17:53.944961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.944386    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568" event={"ID":"7803c74d-04f3-46b2-b0e8-552e5ad89b6c","Type":"ContainerDied","Data":"9bb473bdf22e5d5dae5ba9dbb42f25f8e93c5695bc5604e35f4aa4499b9c8aa7"}
Oct 30 01:17:53.944961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.944423    5104 scope.go:122] "RemoveContainer" containerID="7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81"
Oct 30 01:17:53.946747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.946611    5104 generic.go:356] "Generic (PLEG): container finished" podID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" containerID="fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164" exitCode=0
Oct 30 01:17:53.946747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.946650    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"
Oct 30 01:17:53.946747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.946701    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" event={"ID":"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8","Type":"ContainerDied","Data":"fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164"}
Oct 30 01:17:53.946747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.946737    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g" event={"ID":"3a747f29-e9f5-455f-9caf-17fb4bb9d1d8","Type":"ContainerDied","Data":"766906a4107b41ec2d9167678ececaed2aa2359eae56a3e69951045172a33e31"}
Oct 30 01:17:53.961349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.961323    5104 scope.go:122] "RemoveContainer" containerID="7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81"
Oct 30 01:17:53.961686 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:53.961663    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81\": container with ID starting with 7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81 not found: ID does not exist" containerID="7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81"
Oct 30 01:17:53.961746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.961696    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81"} err="failed to get container status \"7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81\": rpc error: code = NotFound desc = could not find container \"7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81\": container with ID starting with 7bcf0937eb2ff4e2a57196d4009a712dc243c5716f49a165f0be5e55c1eeba81 not found: ID does not exist"
Oct 30 01:17:53.961746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:53.961712    5104 scope.go:122] "RemoveContainer" containerID="fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164"
Oct 30 01:17:54.017481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.006745    5104 scope.go:122] "RemoveContainer" containerID="fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164"
Oct 30 01:17:54.055483 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:17:54.052632    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164\": container with ID starting with fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164 not found: ID does not exist" containerID="fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164"
Oct 30 01:17:54.055483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.052693    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164"} err="failed to get container status \"fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164\": rpc error: code = NotFound desc = could not find container \"fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164\": container with ID starting with fe722acf89029e8f09f5b9d3d22d9ee7e3b8c81d01714e756e2cba83bd1d2164 not found: ID does not exist"
Oct 30 01:17:54.129484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.128785    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"]
Oct 30 01:17:54.139903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.139862    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-34e896/test-model-34e896-kserve-859b999c49-dtw7g"]
Oct 30 01:17:54.146040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.146011    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"]
Oct 30 01:17:54.152720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.152692    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-687471-74d85f5bdd-qw568"]
Oct 30 01:17:54.620234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.620191    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" path="/var/lib/kubelet/pods/3a747f29-e9f5-455f-9caf-17fb4bb9d1d8/volumes"
Oct 30 01:17:54.622514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.621392    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7803c74d-04f3-46b2-b0e8-552e5ad89b6c" path="/var/lib/kubelet/pods/7803c74d-04f3-46b2-b0e8-552e5ad89b6c/volumes"
Oct 30 01:17:54.668870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.668834    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"]
Oct 30 01:17:54.720801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.720765    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"]
Oct 30 01:17:54.722171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.722138    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7803c74d-04f3-46b2-b0e8-552e5ad89b6c" containerName="payload-processing"
Oct 30 01:17:54.722171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.722170    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7803c74d-04f3-46b2-b0e8-552e5ad89b6c" containerName="payload-processing"
Oct 30 01:17:54.722406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.722237    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" containerName="main"
Oct 30 01:17:54.722406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.722245    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" containerName="main"
Oct 30 01:17:54.722538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.722483    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7803c74d-04f3-46b2-b0e8-552e5ad89b6c" containerName="payload-processing"
Oct 30 01:17:54.722538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.722510    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a747f29-e9f5-455f-9caf-17fb4bb9d1d8" containerName="main"
Oct 30 01:17:54.734508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.734471    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:54.741231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.741198    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"]
Oct 30 01:17:54.743678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.742752    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgnc\" (UniqueName: \"kubernetes.io/projected/0a4283b9-51c1-4278-b16d-ad517abf10a9-kube-api-access-9kgnc\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:54.743678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.742846    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:54.743678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.742879    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:54.812481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.755816    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:54.812481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.756029    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:17:54.812481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.756102    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:54.812481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:54.756766    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:55.053821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.043934    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgnc\" (UniqueName: \"kubernetes.io/projected/0a4283b9-51c1-4278-b16d-ad517abf10a9-kube-api-access-9kgnc\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:55.053821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.044168    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:55.053821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.044245    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:55.053821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.045488    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:55.062005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.061966    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 01:17:55.062197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.062040    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 01:17:55.094328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.081210    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgnc\" (UniqueName: \"kubernetes.io/projected/0a4283b9-51c1-4278-b16d-ad517abf10a9-kube-api-access-9kgnc\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:55.116580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.116527    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab03cb2a-pn25v\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:55.195481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.184890    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"]
Oct 30 01:17:55.195481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.194716    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.211514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.211483    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"]
Oct 30 01:17:55.247696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.247659    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.247863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.247732    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hctv\" (UniqueName: \"kubernetes.io/projected/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-kube-api-access-5hctv\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.247949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.247927    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.349081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.348978    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.349260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.349222    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.349328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.349267    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hctv\" (UniqueName: \"kubernetes.io/projected/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-kube-api-access-5hctv\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.349913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.349891    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.351413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.351394    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.354476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.354430    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:55.371450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.370749    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hctv\" (UniqueName: \"kubernetes.io/projected/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-kube-api-access-5hctv\") pod \"maas-api-revoke-sub-ecdd3d36-vgc4v\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.525029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.524992    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:55.718953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.718920    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"]
Oct 30 01:17:55.722366 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:17:55.722332    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4283b9_51c1_4278_b16d_ad517abf10a9.slice/crio-746800e2b0a8100f1c1647664c7736e6e5759142bed6aa2db95e9870bcd735e4 WatchSource:0}: Error finding container 746800e2b0a8100f1c1647664c7736e6e5759142bed6aa2db95e9870bcd735e4: Status 404 returned error can't find the container with id 746800e2b0a8100f1c1647664c7736e6e5759142bed6aa2db95e9870bcd735e4
Oct 30 01:17:55.816553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.810593    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 01:17:55.816553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:55.810659    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 01:17:56.100482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.094276    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v" event={"ID":"0a4283b9-51c1-4278-b16d-ad517abf10a9","Type":"ContainerStarted","Data":"746800e2b0a8100f1c1647664c7736e6e5759142bed6aa2db95e9870bcd735e4"}
Oct 30 01:17:56.111928 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:17:56.111895    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892961d2_a1b7_4a4d_9a82_47faf0d31c4e.slice/crio-efa5e3b606d6eb04f659def72b715972cd2ad11c032d474bfe7be65c2ac281c9 WatchSource:0}: Error finding container efa5e3b606d6eb04f659def72b715972cd2ad11c032d474bfe7be65c2ac281c9: Status 404 returned error can't find the container with id efa5e3b606d6eb04f659def72b715972cd2ad11c032d474bfe7be65c2ac281c9
Oct 30 01:17:56.114229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.114202    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"]
Oct 30 01:17:56.773632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.738408    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"]
Oct 30 01:17:56.873560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.806953    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 01:17:56.873560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.807018    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 01:17:56.904499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.894339    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:56.904499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.894244    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"]
Oct 30 01:17:56.949308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.949266    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:56.949526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.949336    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrc5\" (UniqueName: \"kubernetes.io/projected/9634e32f-9b94-4b22-a347-a36089eaa5a5-kube-api-access-jbrc5\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:56.949526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:56.949508    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.050719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.050626    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.050873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.050832    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.050926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.050888    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrc5\" (UniqueName: \"kubernetes.io/projected/9634e32f-9b94-4b22-a347-a36089eaa5a5-kube-api-access-jbrc5\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.051543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.051518    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.053243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.053219    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.069887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.069859    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrc5\" (UniqueName: \"kubernetes.io/projected/9634e32f-9b94-4b22-a347-a36089eaa5a5-kube-api-access-jbrc5\") pod \"maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.116377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.116340    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v" event={"ID":"0a4283b9-51c1-4278-b16d-ad517abf10a9","Type":"ContainerStarted","Data":"e7a17f517069198fe3edb78dc5ee1f04882f166297f11789fd1e8235b836f32d"}
Oct 30 01:17:57.118407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.118379    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v" event={"ID":"892961d2-a1b7-4a4d-9a82-47faf0d31c4e","Type":"ContainerStarted","Data":"50416af5137c17b57fb0d3b27d33c68bc8f20a061e8820dc00051dc07312017a"}
Oct 30 01:17:57.118561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.118430    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v" event={"ID":"892961d2-a1b7-4a4d-9a82-47faf0d31c4e","Type":"ContainerStarted","Data":"efa5e3b606d6eb04f659def72b715972cd2ad11c032d474bfe7be65c2ac281c9"}
Oct 30 01:17:57.154197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.154145    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v" podStartSLOduration=3.154132145 podStartE2EDuration="3.154132145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:57.153062274 +0000 UTC m=+2118.700752770" watchObservedRunningTime="2026-10-30 01:17:57.154132145 +0000 UTC m=+2118.701822641"
Oct 30 01:17:57.208548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.208481    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v" podStartSLOduration=2.20843994 podStartE2EDuration="2.20843994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:57.191849004 +0000 UTC m=+2118.739539516" watchObservedRunningTime="2026-10-30 01:17:57.20843994 +0000 UTC m=+2118.756130436"
Oct 30 01:17:57.239640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.239606    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:17:57.630371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.630336    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"]
Oct 30 01:17:57.642058 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:17:57.642019    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9634e32f_9b94_4b22_a347_a36089eaa5a5.slice/crio-f58accc08d587a77823c90fa7360ef275fa29b5225b6bc13101b718290e28b38 WatchSource:0}: Error finding container f58accc08d587a77823c90fa7360ef275fa29b5225b6bc13101b718290e28b38: Status 404 returned error can't find the container with id f58accc08d587a77823c90fa7360ef275fa29b5225b6bc13101b718290e28b38
Oct 30 01:17:57.804016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.803914    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 01:17:57.804016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:57.803989    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 01:17:58.132811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:58.132772    5104 generic.go:356] "Generic (PLEG): container finished" podID="0a4283b9-51c1-4278-b16d-ad517abf10a9" containerID="e7a17f517069198fe3edb78dc5ee1f04882f166297f11789fd1e8235b836f32d" exitCode=0
Oct 30 01:17:58.133323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:58.133060    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v" event={"ID":"0a4283b9-51c1-4278-b16d-ad517abf10a9","Type":"ContainerDied","Data":"e7a17f517069198fe3edb78dc5ee1f04882f166297f11789fd1e8235b836f32d"}
Oct 30 01:17:58.135699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:58.135673    5104 generic.go:356] "Generic (PLEG): container finished" podID="892961d2-a1b7-4a4d-9a82-47faf0d31c4e" containerID="50416af5137c17b57fb0d3b27d33c68bc8f20a061e8820dc00051dc07312017a" exitCode=0
Oct 30 01:17:58.135867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:58.135847    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v" event={"ID":"892961d2-a1b7-4a4d-9a82-47faf0d31c4e","Type":"ContainerDied","Data":"50416af5137c17b57fb0d3b27d33c68bc8f20a061e8820dc00051dc07312017a"}
Oct 30 01:17:58.137714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:58.137680    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh" event={"ID":"9634e32f-9b94-4b22-a347-a36089eaa5a5","Type":"ContainerStarted","Data":"f58accc08d587a77823c90fa7360ef275fa29b5225b6bc13101b718290e28b38"}
Oct 30 01:17:59.026675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.026637    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:59.027351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.027318    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:59.028790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.028768    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:17:59.151952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.151860    5104 generic.go:356] "Generic (PLEG): container finished" podID="9634e32f-9b94-4b22-a347-a36089eaa5a5" containerID="0121ce474ef8a478a38df8b5cb855d81cfa5664100335021580e3df7b396b782" exitCode=0
Oct 30 01:17:59.152413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.152056    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh" event={"ID":"9634e32f-9b94-4b22-a347-a36089eaa5a5","Type":"ContainerDied","Data":"0121ce474ef8a478a38df8b5cb855d81cfa5664100335021580e3df7b396b782"}
Oct 30 01:17:59.620339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.620309    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:17:59.623801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.623773    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:17:59.660369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.660337    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-xks-ca\") pod \"0a4283b9-51c1-4278-b16d-ad517abf10a9\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") "
Oct 30 01:17:59.660558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.660406    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-kube-api-access-5hctv\" (UniqueName: \"kubernetes.io/projected/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-kube-api-access-5hctv\") pod \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") "
Oct 30 01:17:59.660642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.660625    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-openshift-ca\") pod \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") "
Oct 30 01:17:59.660818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.660797    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4283b9-51c1-4278-b16d-ad517abf10a9-kube-api-access-9kgnc\" (UniqueName: \"kubernetes.io/projected/0a4283b9-51c1-4278-b16d-ad517abf10a9-kube-api-access-9kgnc\") pod \"0a4283b9-51c1-4278-b16d-ad517abf10a9\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") "
Oct 30 01:17:59.660934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.660860    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-openshift-ca\") pod \"0a4283b9-51c1-4278-b16d-ad517abf10a9\" (UID: \"0a4283b9-51c1-4278-b16d-ad517abf10a9\") "
Oct 30 01:17:59.660934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.660917    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-xks-ca\") pod \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\" (UID: \"892961d2-a1b7-4a4d-9a82-47faf0d31c4e\") "
Oct 30 01:17:59.661120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.661029    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-openshift-ca" pod "892961d2-a1b7-4a4d-9a82-47faf0d31c4e" (UID: "892961d2-a1b7-4a4d-9a82-47faf0d31c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:59.661873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.661353    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-openshift-ca" pod "0a4283b9-51c1-4278-b16d-ad517abf10a9" (UID: "0a4283b9-51c1-4278-b16d-ad517abf10a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:59.661873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.661812    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:59.661873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.661836    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:59.664139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.664112    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-xks-ca" pod "892961d2-a1b7-4a4d-9a82-47faf0d31c4e" (UID: "892961d2-a1b7-4a4d-9a82-47faf0d31c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:59.664875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.664835    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-xks-ca" pod "0a4283b9-51c1-4278-b16d-ad517abf10a9" (UID: "0a4283b9-51c1-4278-b16d-ad517abf10a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:59.665546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.665520    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-kube-api-access-5hctv" pod "892961d2-a1b7-4a4d-9a82-47faf0d31c4e" (UID: "892961d2-a1b7-4a4d-9a82-47faf0d31c4e"). InnerVolumeSpecName "kube-api-access-5hctv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:59.667654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.667618    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4283b9-51c1-4278-b16d-ad517abf10a9-kube-api-access-9kgnc" pod "0a4283b9-51c1-4278-b16d-ad517abf10a9" (UID: "0a4283b9-51c1-4278-b16d-ad517abf10a9"). InnerVolumeSpecName "kube-api-access-9kgnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:59.763855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.763815    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgnc\" (UniqueName: \"kubernetes.io/projected/0a4283b9-51c1-4278-b16d-ad517abf10a9-kube-api-access-9kgnc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:59.763855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.763854    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:59.764076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.763869    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4283b9-51c1-4278-b16d-ad517abf10a9-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:17:59.764076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:17:59.763880    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hctv\" (UniqueName: \"kubernetes.io/projected/892961d2-a1b7-4a4d-9a82-47faf0d31c4e-kube-api-access-5hctv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:00.163250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.163219    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"
Oct 30 01:18:00.163759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.163248    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v" event={"ID":"0a4283b9-51c1-4278-b16d-ad517abf10a9","Type":"ContainerDied","Data":"746800e2b0a8100f1c1647664c7736e6e5759142bed6aa2db95e9870bcd735e4"}
Oct 30 01:18:00.163759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.163301    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746800e2b0a8100f1c1647664c7736e6e5759142bed6aa2db95e9870bcd735e4"
Oct 30 01:18:00.165334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.165297    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v" event={"ID":"892961d2-a1b7-4a4d-9a82-47faf0d31c4e","Type":"ContainerDied","Data":"efa5e3b606d6eb04f659def72b715972cd2ad11c032d474bfe7be65c2ac281c9"}
Oct 30 01:18:00.165507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.165368    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efa5e3b606d6eb04f659def72b715972cd2ad11c032d474bfe7be65c2ac281c9"
Oct 30 01:18:00.165507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.165316    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"
Oct 30 01:18:00.386465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.386423    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:18:00.479885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.479852    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-openshift-ca\") pod \"9634e32f-9b94-4b22-a347-a36089eaa5a5\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") "
Oct 30 01:18:00.480199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.480177    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-xks-ca\") pod \"9634e32f-9b94-4b22-a347-a36089eaa5a5\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") "
Oct 30 01:18:00.480343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.480254    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-openshift-ca" pod "9634e32f-9b94-4b22-a347-a36089eaa5a5" (UID: "9634e32f-9b94-4b22-a347-a36089eaa5a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:00.480343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.480315    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9634e32f-9b94-4b22-a347-a36089eaa5a5-kube-api-access-jbrc5\" (UniqueName: \"kubernetes.io/projected/9634e32f-9b94-4b22-a347-a36089eaa5a5-kube-api-access-jbrc5\") pod \"9634e32f-9b94-4b22-a347-a36089eaa5a5\" (UID: \"9634e32f-9b94-4b22-a347-a36089eaa5a5\") "
Oct 30 01:18:00.481080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.481058    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:00.483212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.483182    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9634e32f-9b94-4b22-a347-a36089eaa5a5-kube-api-access-jbrc5" pod "9634e32f-9b94-4b22-a347-a36089eaa5a5" (UID: "9634e32f-9b94-4b22-a347-a36089eaa5a5"). InnerVolumeSpecName "kube-api-access-jbrc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:00.483578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.483553    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-xks-ca" pod "9634e32f-9b94-4b22-a347-a36089eaa5a5" (UID: "9634e32f-9b94-4b22-a347-a36089eaa5a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:00.582567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.582527    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbrc5\" (UniqueName: \"kubernetes.io/projected/9634e32f-9b94-4b22-a347-a36089eaa5a5-kube-api-access-jbrc5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:00.582567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:00.582565    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634e32f-9b94-4b22-a347-a36089eaa5a5-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:01.176733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:01.176694    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"
Oct 30 01:18:01.176733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:01.176712    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh" event={"ID":"9634e32f-9b94-4b22-a347-a36089eaa5a5","Type":"ContainerDied","Data":"f58accc08d587a77823c90fa7360ef275fa29b5225b6bc13101b718290e28b38"}
Oct 30 01:18:01.177288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:01.176765    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f58accc08d587a77823c90fa7360ef275fa29b5225b6bc13101b718290e28b38"
Oct 30 01:18:07.134904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:07.134871    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"]
Oct 30 01:18:07.168989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:07.168942    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"]
Oct 30 01:18:09.105638 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:09.105597    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:18:09.782251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:09.782173    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82" podUID="fd630c4e-e0f2-4553-985f-94d87184f054" containerName="main" containerID="cri-o://e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85" gracePeriod=45
Oct 30 01:18:10.110230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.110203    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:18:10.200837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.200798    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-tmp-dir\") pod \"fd630c4e-e0f2-4553-985f-94d87184f054\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") "
Oct 30 01:18:10.201072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.200904    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd630c4e-e0f2-4553-985f-94d87184f054-kube-api-access-dqwdp\" (UniqueName: \"kubernetes.io/projected/fd630c4e-e0f2-4553-985f-94d87184f054-kube-api-access-dqwdp\") pod \"fd630c4e-e0f2-4553-985f-94d87184f054\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") "
Oct 30 01:18:10.201072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.201015    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd630c4e-e0f2-4553-985f-94d87184f054-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd630c4e-e0f2-4553-985f-94d87184f054-tls-certs\") pod \"fd630c4e-e0f2-4553-985f-94d87184f054\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") "
Oct 30 01:18:10.201072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.201070    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-home\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-home\") pod \"fd630c4e-e0f2-4553-985f-94d87184f054\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") "
Oct 30 01:18:10.201269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.201109    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-model-cache\") pod \"fd630c4e-e0f2-4553-985f-94d87184f054\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") "
Oct 30 01:18:10.201269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.201108    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-tmp-dir" pod "fd630c4e-e0f2-4553-985f-94d87184f054" (UID: "fd630c4e-e0f2-4553-985f-94d87184f054"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:10.201391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.201371    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-dshm\") pod \"fd630c4e-e0f2-4553-985f-94d87184f054\" (UID: \"fd630c4e-e0f2-4553-985f-94d87184f054\") "
Oct 30 01:18:10.201719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.201657    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-home" pod "fd630c4e-e0f2-4553-985f-94d87184f054" (UID: "fd630c4e-e0f2-4553-985f-94d87184f054"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:10.201948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.201907    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-model-cache" pod "fd630c4e-e0f2-4553-985f-94d87184f054" (UID: "fd630c4e-e0f2-4553-985f-94d87184f054"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:10.202703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.202675    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:10.202825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.202708    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:10.202825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.202723    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:10.204565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.204420    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd630c4e-e0f2-4553-985f-94d87184f054-tls-certs" pod "fd630c4e-e0f2-4553-985f-94d87184f054" (UID: "fd630c4e-e0f2-4553-985f-94d87184f054"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:10.205102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.205074    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd630c4e-e0f2-4553-985f-94d87184f054-kube-api-access-dqwdp" pod "fd630c4e-e0f2-4553-985f-94d87184f054" (UID: "fd630c4e-e0f2-4553-985f-94d87184f054"). InnerVolumeSpecName "kube-api-access-dqwdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:10.205205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.205166    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-dshm" pod "fd630c4e-e0f2-4553-985f-94d87184f054" (UID: "fd630c4e-e0f2-4553-985f-94d87184f054"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:10.287358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.287320    5104 generic.go:356] "Generic (PLEG): container finished" podID="fd630c4e-e0f2-4553-985f-94d87184f054" containerID="e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85" exitCode=0
Oct 30 01:18:10.287596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.287382    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"
Oct 30 01:18:10.287596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.287541    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82" event={"ID":"fd630c4e-e0f2-4553-985f-94d87184f054","Type":"ContainerDied","Data":"e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85"}
Oct 30 01:18:10.287733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.287631    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82" event={"ID":"fd630c4e-e0f2-4553-985f-94d87184f054","Type":"ContainerDied","Data":"dd0d02807d66b5dfce56fd52c3d586968ec24c999a2e1bcd9139e1990fdfd788"}
Oct 30 01:18:10.287733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.287659    5104 scope.go:122] "RemoveContainer" containerID="e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85"
Oct 30 01:18:10.301500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.301435    5104 scope.go:122] "RemoveContainer" containerID="e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85"
Oct 30 01:18:10.301813 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:10.301792    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85\": container with ID starting with e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85 not found: ID does not exist" containerID="e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85"
Oct 30 01:18:10.301922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.301823    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85"} err="failed to get container status \"e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85\": rpc error: code = NotFound desc = could not find container \"e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85\": container with ID starting with e366dcc8adf00c999f9e1ab252615156f772e231401d9a4d76b8f603cc145e85 not found: ID does not exist"
Oct 30 01:18:10.304049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.304029    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqwdp\" (UniqueName: \"kubernetes.io/projected/fd630c4e-e0f2-4553-985f-94d87184f054-kube-api-access-dqwdp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:10.304149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.304050    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd630c4e-e0f2-4553-985f-94d87184f054-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:10.304149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.304062    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd630c4e-e0f2-4553-985f-94d87184f054-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:10.339422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.339336    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"]
Oct 30 01:18:10.344954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.344924    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-82e9bf/test-model-82e9bf-kserve-68db7b8cf-q2n82"]
Oct 30 01:18:10.584590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:10.584552    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd630c4e-e0f2-4553-985f-94d87184f054" path="/var/lib/kubelet/pods/fd630c4e-e0f2-4553-985f-94d87184f054/volumes"
Oct 30 01:18:12.135173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:12.135133    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" podUID="b950c829-f560-490a-8265-f6ef547abfee" containerName="payload-processing" containerID="cri-o://a36f637b90fca3c6ac4585d4fbb892926915d19e915dfc07fb6591ce45392c85" gracePeriod=25
Oct 30 01:18:12.169981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:12.169939    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" podUID="b02c7785-9248-485d-8d94-23290c4f6c0d" containerName="payload-pre-processing" containerID="cri-o://e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f" gracePeriod=25
Oct 30 01:18:15.381971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:15.381929    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" podUID="b02c7785-9248-485d-8d94-23290c4f6c0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:15.382531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:15.381936    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" podUID="b950c829-f560-490a-8265-f6ef547abfee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:17.182077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:17.182039    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:18:19.228763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:19.228717    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"]
Oct 30 01:18:19.302702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:19.300350    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e92dab-fvpbh"]
Oct 30 01:18:20.592280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:20.592246    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9634e32f-9b94-4b22-a347-a36089eaa5a5" path="/var/lib/kubelet/pods/9634e32f-9b94-4b22-a347-a36089eaa5a5/volumes"
Oct 30 01:18:21.323585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.323544    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"]
Oct 30 01:18:21.336390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.336357    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-687471-q2s8z"]
Oct 30 01:18:21.468879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.468842    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"]
Oct 30 01:18:21.470372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.470347    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0a4283b9-51c1-4278-b16d-ad517abf10a9" containerName="revoke-keys"
Oct 30 01:18:21.470554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.470538    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4283b9-51c1-4278-b16d-ad517abf10a9" containerName="revoke-keys"
Oct 30 01:18:21.470682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.470672    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9634e32f-9b94-4b22-a347-a36089eaa5a5" containerName="revoke-keys"
Oct 30 01:18:21.470765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.470754    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9634e32f-9b94-4b22-a347-a36089eaa5a5" containerName="revoke-keys"
Oct 30 01:18:21.470856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.470846    5104 cpu_manager.go:414] "containerMap: removing container" podUID="892961d2-a1b7-4a4d-9a82-47faf0d31c4e" containerName="revoke-keys"
Oct 30 01:18:21.470930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.470921    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892961d2-a1b7-4a4d-9a82-47faf0d31c4e" containerName="revoke-keys"
Oct 30 01:18:21.471037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.471028    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fd630c4e-e0f2-4553-985f-94d87184f054" containerName="main"
Oct 30 01:18:21.471109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.471100    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd630c4e-e0f2-4553-985f-94d87184f054" containerName="main"
Oct 30 01:18:21.471429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.471405    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4283b9-51c1-4278-b16d-ad517abf10a9" containerName="revoke-keys"
Oct 30 01:18:21.471569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.471557    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9634e32f-9b94-4b22-a347-a36089eaa5a5" containerName="revoke-keys"
Oct 30 01:18:21.471663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.471653    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="892961d2-a1b7-4a4d-9a82-47faf0d31c4e" containerName="revoke-keys"
Oct 30 01:18:21.471774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.471762    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd630c4e-e0f2-4553-985f-94d87184f054" containerName="main"
Oct 30 01:18:21.478617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.478592    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.495571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.495550    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"openshift-service-ca.crt\""
Oct 30 01:18:21.495725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.495613    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"default-dockercfg-tx6rt\""
Oct 30 01:18:21.495725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.495692    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"rate-test-model-69dc90-kserve-self-signed-certs\""
Oct 30 01:18:21.495846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.495764    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-69dc90\"/\"kube-root-ca.crt\""
Oct 30 01:18:21.497736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.497714    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"]
Oct 30 01:18:21.578313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.578222    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-tmp-dir\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.578313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.578272    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-dshm\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.578649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.578623    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6wp\" (UniqueName: \"kubernetes.io/projected/731562c2-d399-49ee-9e12-cecfb5133e6f-kube-api-access-hj6wp\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.578729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.578697    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-home\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.578808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.578755    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731562c2-d399-49ee-9e12-cecfb5133e6f-tls-certs\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.578865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.578809    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-model-cache\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.702147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.696736    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-model-cache\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.702147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.696879    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-tmp-dir\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.702147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.696929    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-dshm\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.702147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.697243    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6wp\" (UniqueName: \"kubernetes.io/projected/731562c2-d399-49ee-9e12-cecfb5133e6f-kube-api-access-hj6wp\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.702147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.697289    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-home\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.702147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.697318    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731562c2-d399-49ee-9e12-cecfb5133e6f-tls-certs\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.723390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.718427    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-model-cache\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.734876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.731640    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-tmp-dir\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.734876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.732353    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731562c2-d399-49ee-9e12-cecfb5133e6f-tls-certs\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.734876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.733491    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-home\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.745286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.744053    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-dshm\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.767361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.767327    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6wp\" (UniqueName: \"kubernetes.io/projected/731562c2-d399-49ee-9e12-cecfb5133e6f-kube-api-access-hj6wp\") pod \"rate-test-model-69dc90-kserve-548b8bb45-prf56\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") " pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:21.805196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:21.805159    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:22.213126 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:22.213083    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"]
Oct 30 01:18:22.584053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:22.583959    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e758f3-e8be-472b-8b20-a69d2f26794e" path="/var/lib/kubelet/pods/28e758f3-e8be-472b-8b20-a69d2f26794e/volumes"
Oct 30 01:18:22.642305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:22.642270    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" event={"ID":"731562c2-d399-49ee-9e12-cecfb5133e6f","Type":"ContainerStarted","Data":"f7a805aee37fc64bcb1cf8d0f93c925daf42bb40e7c4722766cae4a2e1325b01"}
Oct 30 01:18:22.642522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:22.642320    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" event={"ID":"731562c2-d399-49ee-9e12-cecfb5133e6f","Type":"ContainerStarted","Data":"daa9c5d3ce1aa623065719377a0e658718add47ac447599295e7dbda3d25c306"}
Oct 30 01:18:22.681926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:22.681871    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podStartSLOduration=1.6818559199999998 podStartE2EDuration="1.68185592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:22.679214817 +0000 UTC m=+2144.226905313" watchObservedRunningTime="2026-10-30 01:18:22.68185592 +0000 UTC m=+2144.229546416"
Oct 30 01:18:22.812484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:22.812292    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:25.181624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:25.181529    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"]
Oct 30 01:18:25.208616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:25.208579    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4dde48-8z8rs"]
Oct 30 01:18:25.431093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:25.430662    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" podUID="b950c829-f560-490a-8265-f6ef547abfee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:25.467686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:25.467590    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" podUID="b02c7785-9248-485d-8d94-23290c4f6c0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:25.934559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:25.934526    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"]
Oct 30 01:18:25.934819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:25.934794    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" containerID="cri-o://51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9" gracePeriod=30
Oct 30 01:18:26.222514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.222386    5104 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:18:26.223150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.223100    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:18:26.426931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.426894    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"]
Oct 30 01:18:26.433234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.433208    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.451027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.451001    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:18:26.451282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.451231    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:18:26.451411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.451288    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:18:26.451499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.451316    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:18:26.452556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.452488    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"]
Oct 30 01:18:26.547853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.547762    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.548037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.548012    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.548093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.548048    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmps6\" (UniqueName: \"kubernetes.io/projected/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-kube-api-access-pmps6\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.583116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.583079    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635f89ba-f80b-4b1f-8205-74a867b78ebf" path="/var/lib/kubelet/pods/635f89ba-f80b-4b1f-8205-74a867b78ebf/volumes"
Oct 30 01:18:26.650167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.650128    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.650373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.650358    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.650441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.650385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmps6\" (UniqueName: \"kubernetes.io/projected/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-kube-api-access-pmps6\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.651138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.651113    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.652779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.652757    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.670099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.670063    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmps6\" (UniqueName: \"kubernetes.io/projected/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-kube-api-access-pmps6\") pod \"maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:26.751083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:26.751046    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:27.062424 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:18:27.062329    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca4b507_869b_400c_9fb4_b9c26f5ec4dc.slice/crio-a9168719c83bf6c4cef02b40fb2c2dc4a8920cccd184e7042b6f87a1b60c39d0 WatchSource:0}: Error finding container a9168719c83bf6c4cef02b40fb2c2dc4a8920cccd184e7042b6f87a1b60c39d0: Status 404 returned error can't find the container with id a9168719c83bf6c4cef02b40fb2c2dc4a8920cccd184e7042b6f87a1b60c39d0
Oct 30 01:18:27.065856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.065830    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"]
Oct 30 01:18:27.647090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.647047    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"]
Oct 30 01:18:27.760544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.759711    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx" event={"ID":"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc","Type":"ContainerStarted","Data":"9e597d44c4a5f0cd35df6d67276e22aa4216e9d750aeb171c6323f4ee46f71be"}
Oct 30 01:18:27.760544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.759795    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx" event={"ID":"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc","Type":"ContainerStarted","Data":"a9168719c83bf6c4cef02b40fb2c2dc4a8920cccd184e7042b6f87a1b60c39d0"}
Oct 30 01:18:27.933500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.931761    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"]
Oct 30 01:18:27.942189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.942149    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:27.950217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.950136    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"]
Oct 30 01:18:27.952894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:27.952688    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx" podStartSLOduration=1.952668896 podStartE2EDuration="1.952668896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:27.805921369 +0000 UTC m=+2149.353611865" watchObservedRunningTime="2026-10-30 01:18:27.952668896 +0000 UTC m=+2149.500359392"
Oct 30 01:18:28.007928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.007858    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"]
Oct 30 01:18:28.036986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.036885    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.037153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.037110    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.037417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.037389    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvgk6\" (UniqueName: \"kubernetes.io/projected/9b5f78e4-398b-4d80-80a1-14377945ee5a-kube-api-access-rvgk6\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.159481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.156522    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.171498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.169513    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgk6\" (UniqueName: \"kubernetes.io/projected/9b5f78e4-398b-4d80-80a1-14377945ee5a-kube-api-access-rvgk6\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.171498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.169696    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.171498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.170750    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.201486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.201220    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvgk6\" (UniqueName: \"kubernetes.io/projected/9b5f78e4-398b-4d80-80a1-14377945ee5a-kube-api-access-rvgk6\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.281564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.267988    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.282585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.282084    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:28.529553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.529518    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"]
Oct 30 01:18:28.536035 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.536007    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.553547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.553448    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"]
Oct 30 01:18:28.604948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.604049    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.604948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.604233    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z259s\" (UniqueName: \"kubernetes.io/projected/843d580a-b98b-4426-8ac7-844bb4ddf3bb-kube-api-access-z259s\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.604948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.604551    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.629379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.629338    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"]
Oct 30 01:18:28.630895 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:18:28.630869    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5f78e4_398b_4d80_80a1_14377945ee5a.slice/crio-ef22bbb1f8b914fdaeae095adba770bc9899e0415324a9b9e0ed2f5e94a377ff WatchSource:0}: Error finding container ef22bbb1f8b914fdaeae095adba770bc9899e0415324a9b9e0ed2f5e94a377ff: Status 404 returned error can't find the container with id ef22bbb1f8b914fdaeae095adba770bc9899e0415324a9b9e0ed2f5e94a377ff
Oct 30 01:18:28.705697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.705661    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.706173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.705816    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.706173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.705940    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z259s\" (UniqueName: \"kubernetes.io/projected/843d580a-b98b-4426-8ac7-844bb4ddf3bb-kube-api-access-z259s\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.706612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.706586    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.708296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.708268    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.734186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.734149    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z259s\" (UniqueName: \"kubernetes.io/projected/843d580a-b98b-4426-8ac7-844bb4ddf3bb-kube-api-access-z259s\") pod \"maas-api-revoke-sub-149f6257-6ddrv\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.775040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.775003    5104 generic.go:356] "Generic (PLEG): container finished" podID="9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" containerID="9e597d44c4a5f0cd35df6d67276e22aa4216e9d750aeb171c6323f4ee46f71be" exitCode=0
Oct 30 01:18:28.775251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.775091    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx" event={"ID":"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc","Type":"ContainerDied","Data":"9e597d44c4a5f0cd35df6d67276e22aa4216e9d750aeb171c6323f4ee46f71be"}
Oct 30 01:18:28.776786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.776761    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn" event={"ID":"9b5f78e4-398b-4d80-80a1-14377945ee5a","Type":"ContainerStarted","Data":"ef22bbb1f8b914fdaeae095adba770bc9899e0415324a9b9e0ed2f5e94a377ff"}
Oct 30 01:18:28.855683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:28.855599    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:28.919622 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:28.919558    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5f78e4_398b_4d80_80a1_14377945ee5a.slice/crio-cef0ba6f7c5e9020ee605b1f38b916c0bd41b3659e2ba91e9a21859839f7536a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:18:29.194400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:29.194371    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"]
Oct 30 01:18:29.198402 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:18:29.198374    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843d580a_b98b_4426_8ac7_844bb4ddf3bb.slice/crio-b83550149a0295f370d246e18b2fecefe7a93c16ecdae76eba6a7caa7c8089cd WatchSource:0}: Error finding container b83550149a0295f370d246e18b2fecefe7a93c16ecdae76eba6a7caa7c8089cd: Status 404 returned error can't find the container with id b83550149a0295f370d246e18b2fecefe7a93c16ecdae76eba6a7caa7c8089cd
Oct 30 01:18:29.367506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:29.365736    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"]
Oct 30 01:18:29.367506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:29.366003    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy" containerID="cri-o://0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f" gracePeriod=30
Oct 30 01:18:29.802398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:29.802096    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv" event={"ID":"843d580a-b98b-4426-8ac7-844bb4ddf3bb","Type":"ContainerStarted","Data":"b83550149a0295f370d246e18b2fecefe7a93c16ecdae76eba6a7caa7c8089cd"}
Oct 30 01:18:29.816045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:29.814204    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn" event={"ID":"9b5f78e4-398b-4d80-80a1-14377945ee5a","Type":"ContainerStarted","Data":"cef0ba6f7c5e9020ee605b1f38b916c0bd41b3659e2ba91e9a21859839f7536a"}
Oct 30 01:18:29.854777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:29.854715    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn" podStartSLOduration=2.854695548 podStartE2EDuration="2.854695548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:29.850911435 +0000 UTC m=+2151.398601945" watchObservedRunningTime="2026-10-30 01:18:29.854695548 +0000 UTC m=+2151.402386045"
Oct 30 01:18:30.120523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.120374    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:30.239036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.238999    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-xks-ca\") pod \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") "
Oct 30 01:18:30.239393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.239364    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-openshift-ca\") pod \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") "
Oct 30 01:18:30.239520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.239447    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-kube-api-access-pmps6\" (UniqueName: \"kubernetes.io/projected/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-kube-api-access-pmps6\") pod \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\" (UID: \"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc\") "
Oct 30 01:18:30.240971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.240940    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-openshift-ca" pod "9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" (UID: "9ca4b507-869b-400c-9fb4-b9c26f5ec4dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:30.247956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.244415    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-kube-api-access-pmps6" pod "9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" (UID: "9ca4b507-869b-400c-9fb4-b9c26f5ec4dc"). InnerVolumeSpecName "kube-api-access-pmps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:30.252986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.252947    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-xks-ca" pod "9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" (UID: "9ca4b507-869b-400c-9fb4-b9c26f5ec4dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:30.342345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.342302    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:30.342345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.342340    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:30.342345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.342351    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmps6\" (UniqueName: \"kubernetes.io/projected/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc-kube-api-access-pmps6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:30.895066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.894349    5104 generic.go:356] "Generic (PLEG): container finished" podID="843d580a-b98b-4426-8ac7-844bb4ddf3bb" containerID="ff2db8057cd1ac2f6ccc1a112ae04b8e0e7bdc61cab7639cb1b20f76c661aa67" exitCode=0
Oct 30 01:18:30.895066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.894526    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv" event={"ID":"843d580a-b98b-4426-8ac7-844bb4ddf3bb","Type":"ContainerDied","Data":"ff2db8057cd1ac2f6ccc1a112ae04b8e0e7bdc61cab7639cb1b20f76c661aa67"}
Oct 30 01:18:30.910962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.910923    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx" event={"ID":"9ca4b507-869b-400c-9fb4-b9c26f5ec4dc","Type":"ContainerDied","Data":"a9168719c83bf6c4cef02b40fb2c2dc4a8920cccd184e7042b6f87a1b60c39d0"}
Oct 30 01:18:30.911204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.911187    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9168719c83bf6c4cef02b40fb2c2dc4a8920cccd184e7042b6f87a1b60c39d0"
Oct 30 01:18:30.911407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.911393    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"
Oct 30 01:18:30.932421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.929634    5104 generic.go:356] "Generic (PLEG): container finished" podID="9b5f78e4-398b-4d80-80a1-14377945ee5a" containerID="cef0ba6f7c5e9020ee605b1f38b916c0bd41b3659e2ba91e9a21859839f7536a" exitCode=0
Oct 30 01:18:30.932899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:30.932864    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn" event={"ID":"9b5f78e4-398b-4d80-80a1-14377945ee5a","Type":"ContainerDied","Data":"cef0ba6f7c5e9020ee605b1f38b916c0bd41b3659e2ba91e9a21859839f7536a"}
Oct 30 01:18:31.489116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.489088    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:18:31.560666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.560635    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-podinfo\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.560861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.560752    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istiod-ca-cert\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.560861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.560797    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-data\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.560956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.560896    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-token\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-token\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.560956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.560938    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-certs\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.561057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.560987    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-kube-api-access-xqqqx\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-kube-api-access-xqqqx\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.561057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.561038    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-credential-socket\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.561313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.561287    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-socket\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.561440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.561307    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-data" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:31.561440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.561352    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-certs" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:31.561678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.561658    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istiod-ca-cert" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.561939    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-socket" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.561981    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-credential-socket" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.562099    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-envoy\") pod \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\" (UID: \"2d47b9bd-4728-4846-80d9-ac2717fd0c74\") "
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.562898    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.562953    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.562968    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.562979    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.563011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.562988    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.563924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.563903    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-podinfo" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:18:31.566932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.566686    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-envoy" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:31.566932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.566865    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-token" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:31.566932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.566896    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-kube-api-access-xqqqx" pod "2d47b9bd-4728-4846-80d9-ac2717fd0c74" (UID: "2d47b9bd-4728-4846-80d9-ac2717fd0c74"). InnerVolumeSpecName "kube-api-access-xqqqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:31.665508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.665439    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.665508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.665508    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.665729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.665523    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.665729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.665538    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqqqx\" (UniqueName: \"kubernetes.io/projected/2d47b9bd-4728-4846-80d9-ac2717fd0c74-kube-api-access-xqqqx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:31.806702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.806612    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:31.820935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.820903    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:31.944737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.944702    5104 generic.go:356] "Generic (PLEG): container finished" podID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerID="51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9" exitCode=0
Oct 30 01:18:31.945224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.944791    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" event={"ID":"2d47b9bd-4728-4846-80d9-ac2717fd0c74","Type":"ContainerDied","Data":"51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9"}
Oct 30 01:18:31.945224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.944887    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk" event={"ID":"2d47b9bd-4728-4846-80d9-ac2717fd0c74","Type":"ContainerDied","Data":"c1622e2b3b5510f7c414d956a8194a967fb3dde585ef44986de75afc4f21321a"}
Oct 30 01:18:31.945224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.944919    5104 scope.go:122] "RemoveContainer" containerID="51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9"
Oct 30 01:18:31.945224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.944919    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"
Oct 30 01:18:31.962200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.962167    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:18:31.964077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.963972    5104 scope.go:122] "RemoveContainer" containerID="51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9"
Oct 30 01:18:31.967424 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:31.967397    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9\": container with ID starting with 51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9 not found: ID does not exist" containerID="51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9"
Oct 30 01:18:31.967613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:31.967439    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9"} err="failed to get container status \"51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9\": rpc error: code = NotFound desc = could not find container \"51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9\": container with ID starting with 51e569eef9d508377fc57342487368e6bfa3f37b935d5f25eba1a53441d7c2c9 not found: ID does not exist"
Oct 30 01:18:32.007201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.007134    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"]
Oct 30 01:18:32.017000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.016962    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-w5vgk"]
Oct 30 01:18:32.341480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.325775    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:32.345481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.342737    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:32.620481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.619481    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5f78e4-398b-4d80-80a1-14377945ee5a-kube-api-access-rvgk6\" (UniqueName: \"kubernetes.io/projected/9b5f78e4-398b-4d80-80a1-14377945ee5a-kube-api-access-rvgk6\") pod \"9b5f78e4-398b-4d80-80a1-14377945ee5a\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") "
Oct 30 01:18:32.620481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.619763    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-openshift-ca\") pod \"9b5f78e4-398b-4d80-80a1-14377945ee5a\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") "
Oct 30 01:18:32.620481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.619811    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-xks-ca\") pod \"9b5f78e4-398b-4d80-80a1-14377945ee5a\" (UID: \"9b5f78e4-398b-4d80-80a1-14377945ee5a\") "
Oct 30 01:18:32.620481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.619887    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-openshift-ca\") pod \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") "
Oct 30 01:18:32.620481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.619929    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-xks-ca\") pod \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") "
Oct 30 01:18:32.620481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.620032    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843d580a-b98b-4426-8ac7-844bb4ddf3bb-kube-api-access-z259s\" (UniqueName: \"kubernetes.io/projected/843d580a-b98b-4426-8ac7-844bb4ddf3bb-kube-api-access-z259s\") pod \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\" (UID: \"843d580a-b98b-4426-8ac7-844bb4ddf3bb\") "
Oct 30 01:18:32.751957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.686260    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5f78e4-398b-4d80-80a1-14377945ee5a-kube-api-access-rvgk6" pod "9b5f78e4-398b-4d80-80a1-14377945ee5a" (UID: "9b5f78e4-398b-4d80-80a1-14377945ee5a"). InnerVolumeSpecName "kube-api-access-rvgk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:32.753484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.753348    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-openshift-ca" pod "9b5f78e4-398b-4d80-80a1-14377945ee5a" (UID: "9b5f78e4-398b-4d80-80a1-14377945ee5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:32.755469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.754285    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-xks-ca" pod "9b5f78e4-398b-4d80-80a1-14377945ee5a" (UID: "9b5f78e4-398b-4d80-80a1-14377945ee5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:32.755469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.754321    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-openshift-ca" pod "843d580a-b98b-4426-8ac7-844bb4ddf3bb" (UID: "843d580a-b98b-4426-8ac7-844bb4ddf3bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:32.755469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.755399    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843d580a-b98b-4426-8ac7-844bb4ddf3bb-kube-api-access-z259s" pod "843d580a-b98b-4426-8ac7-844bb4ddf3bb" (UID: "843d580a-b98b-4426-8ac7-844bb4ddf3bb"). InnerVolumeSpecName "kube-api-access-z259s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:32.788706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.777072    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" path="/var/lib/kubelet/pods/2d47b9bd-4728-4846-80d9-ac2717fd0c74/volumes"
Oct 30 01:18:32.788706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.779574    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-xks-ca" pod "843d580a-b98b-4426-8ac7-844bb4ddf3bb" (UID: "843d580a-b98b-4426-8ac7-844bb4ddf3bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:32.829502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.824800    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"]
Oct 30 01:18:32.829502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.825044    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" containerID="cri-o://d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6" gracePeriod=30
Oct 30 01:18:32.907738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.907159    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:18:32.937492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.934178    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02c7785-9248-485d-8d94-23290c4f6c0d-kube-api-access-sv7gp\" (UniqueName: \"kubernetes.io/projected/b02c7785-9248-485d-8d94-23290c4f6c0d-kube-api-access-sv7gp\") pod \"b02c7785-9248-485d-8d94-23290c4f6c0d\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") "
Oct 30 01:18:32.950297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.949061    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02c7785-9248-485d-8d94-23290c4f6c0d-kube-api-access-sv7gp" pod "b02c7785-9248-485d-8d94-23290c4f6c0d" (UID: "b02c7785-9248-485d-8d94-23290c4f6c0d"). InnerVolumeSpecName "kube-api-access-sv7gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.959690    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02c7785-9248-485d-8d94-23290c4f6c0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02c7785-9248-485d-8d94-23290c4f6c0d-plugins-config-volume\") pod \"b02c7785-9248-485d-8d94-23290c4f6c0d\" (UID: \"b02c7785-9248-485d-8d94-23290c4f6c0d\") "
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.960864    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv7gp\" (UniqueName: \"kubernetes.io/projected/b02c7785-9248-485d-8d94-23290c4f6c0d-kube-api-access-sv7gp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.960893    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.960910    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5f78e4-398b-4d80-80a1-14377945ee5a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.960925    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.960939    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d580a-b98b-4426-8ac7-844bb4ddf3bb-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.960967    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z259s\" (UniqueName: \"kubernetes.io/projected/843d580a-b98b-4426-8ac7-844bb4ddf3bb-kube-api-access-z259s\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.961490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.960981    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvgk6\" (UniqueName: \"kubernetes.io/projected/9b5f78e4-398b-4d80-80a1-14377945ee5a-kube-api-access-rvgk6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:32.980474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.972598    5104 generic.go:356] "Generic (PLEG): container finished" podID="b950c829-f560-490a-8265-f6ef547abfee" containerID="a36f637b90fca3c6ac4585d4fbb892926915d19e915dfc07fb6591ce45392c85" exitCode=0
Oct 30 01:18:32.980474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:32.977904    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" event={"ID":"b950c829-f560-490a-8265-f6ef547abfee","Type":"ContainerDied","Data":"a36f637b90fca3c6ac4585d4fbb892926915d19e915dfc07fb6591ce45392c85"}
Oct 30 01:18:33.113057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.096584    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02c7785-9248-485d-8d94-23290c4f6c0d-plugins-config-volume" pod "b02c7785-9248-485d-8d94-23290c4f6c0d" (UID: "b02c7785-9248-485d-8d94-23290c4f6c0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:33.113057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.106530    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02c7785-9248-485d-8d94-23290c4f6c0d-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:33.157255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.143854    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn" event={"ID":"9b5f78e4-398b-4d80-80a1-14377945ee5a","Type":"ContainerDied","Data":"ef22bbb1f8b914fdaeae095adba770bc9899e0415324a9b9e0ed2f5e94a377ff"}
Oct 30 01:18:33.157255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.144015    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef22bbb1f8b914fdaeae095adba770bc9899e0415324a9b9e0ed2f5e94a377ff"
Oct 30 01:18:33.163655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.163621    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"
Oct 30 01:18:33.242192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.242150    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"]
Oct 30 01:18:33.243508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243478    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy"
Oct 30 01:18:33.243508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243506    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243544    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" containerName="revoke-keys"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243552    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" containerName="revoke-keys"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243564    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b02c7785-9248-485d-8d94-23290c4f6c0d" containerName="payload-pre-processing"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243572    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02c7785-9248-485d-8d94-23290c4f6c0d" containerName="payload-pre-processing"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243583    5104 cpu_manager.go:414] "containerMap: removing container" podUID="843d580a-b98b-4426-8ac7-844bb4ddf3bb" containerName="revoke-keys"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243590    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843d580a-b98b-4426-8ac7-844bb4ddf3bb" containerName="revoke-keys"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243647    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9b5f78e4-398b-4d80-80a1-14377945ee5a" containerName="revoke-keys"
Oct 30 01:18:33.243744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243655    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5f78e4-398b-4d80-80a1-14377945ee5a" containerName="revoke-keys"
Oct 30 01:18:33.244151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243937    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" containerName="revoke-keys"
Oct 30 01:18:33.244151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243955    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02c7785-9248-485d-8d94-23290c4f6c0d" containerName="payload-pre-processing"
Oct 30 01:18:33.244151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243966    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="843d580a-b98b-4426-8ac7-844bb4ddf3bb" containerName="revoke-keys"
Oct 30 01:18:33.244151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.243985    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5f78e4-398b-4d80-80a1-14377945ee5a" containerName="revoke-keys"
Oct 30 01:18:33.244151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.244000    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d47b9bd-4728-4846-80d9-ac2717fd0c74" containerName="istio-proxy"
Oct 30 01:18:33.249020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.248996    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.250298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.250277    5104 generic.go:356] "Generic (PLEG): container finished" podID="b02c7785-9248-485d-8d94-23290c4f6c0d" containerID="e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f" exitCode=0
Oct 30 01:18:33.250675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.250659    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"
Oct 30 01:18:33.256797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.256761    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" event={"ID":"b02c7785-9248-485d-8d94-23290c4f6c0d","Type":"ContainerDied","Data":"e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f"}
Oct 30 01:18:33.256954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.256834    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc" event={"ID":"b02c7785-9248-485d-8d94-23290c4f6c0d","Type":"ContainerDied","Data":"34e9ba98532904601a02cf55b21193a93de6befeca81912b27b083c528182e95"}
Oct 30 01:18:33.256954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.256869    5104 scope.go:122] "RemoveContainer" containerID="e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f"
Oct 30 01:18:33.261287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.261260    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"]
Oct 30 01:18:33.270506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.268967    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"
Oct 30 01:18:33.270506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.268966    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv" event={"ID":"843d580a-b98b-4426-8ac7-844bb4ddf3bb","Type":"ContainerDied","Data":"b83550149a0295f370d246e18b2fecefe7a93c16ecdae76eba6a7caa7c8089cd"}
Oct 30 01:18:33.270506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.269085    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b83550149a0295f370d246e18b2fecefe7a93c16ecdae76eba6a7caa7c8089cd"
Oct 30 01:18:33.303725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.299752    5104 scope.go:122] "RemoveContainer" containerID="e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f"
Oct 30 01:18:33.303725 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:33.300266    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f\": container with ID starting with e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f not found: ID does not exist" containerID="e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f"
Oct 30 01:18:33.303725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.300302    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f"} err="failed to get container status \"e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f\": rpc error: code = NotFound desc = could not find container \"e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f\": container with ID starting with e99628a90cf41824d2ae2c00f8b46ccc7dcde26a4c657f905ff198c794ef734f not found: ID does not exist"
Oct 30 01:18:33.331786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.331738    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"]
Oct 30 01:18:33.372581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.372441    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e92dab-dbcfdfcc5-2nsjc"]
Oct 30 01:18:33.385598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.377827    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:18:33.415479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.415030    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.415479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.415162    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z689r\" (UniqueName: \"kubernetes.io/projected/86d9557d-48f8-409a-9cbc-22b8a39739c3-kube-api-access-z689r\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.416511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.415931    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.516750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.516713    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b950c829-f560-490a-8265-f6ef547abfee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b950c829-f560-490a-8265-f6ef547abfee-plugins-config-volume\") pod \"b950c829-f560-490a-8265-f6ef547abfee\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") "
Oct 30 01:18:33.516941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.516922    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b950c829-f560-490a-8265-f6ef547abfee-kube-api-access-9rstb\" (UniqueName: \"kubernetes.io/projected/b950c829-f560-490a-8265-f6ef547abfee-kube-api-access-9rstb\") pod \"b950c829-f560-490a-8265-f6ef547abfee\" (UID: \"b950c829-f560-490a-8265-f6ef547abfee\") "
Oct 30 01:18:33.517159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.517133    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b950c829-f560-490a-8265-f6ef547abfee-plugins-config-volume" pod "b950c829-f560-490a-8265-f6ef547abfee" (UID: "b950c829-f560-490a-8265-f6ef547abfee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:33.517439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.517419    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.517629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.517610    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.517712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.517692    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z689r\" (UniqueName: \"kubernetes.io/projected/86d9557d-48f8-409a-9cbc-22b8a39739c3-kube-api-access-z689r\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.517979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.517846    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b950c829-f560-490a-8265-f6ef547abfee-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:33.518445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.518418    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.519434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.519412    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b950c829-f560-490a-8265-f6ef547abfee-kube-api-access-9rstb" pod "b950c829-f560-490a-8265-f6ef547abfee" (UID: "b950c829-f560-490a-8265-f6ef547abfee"). InnerVolumeSpecName "kube-api-access-9rstb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:33.520233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.520211    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.537389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.537350    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z689r\" (UniqueName: \"kubernetes.io/projected/86d9557d-48f8-409a-9cbc-22b8a39739c3-kube-api-access-z689r\") pod \"maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.619314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.619270    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rstb\" (UniqueName: \"kubernetes.io/projected/b950c829-f560-490a-8265-f6ef547abfee-kube-api-access-9rstb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:33.628969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.628874    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:33.943201 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:18:33.943165    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d9557d_48f8_409a_9cbc_22b8a39739c3.slice/crio-7846af4c430cbe18afc7856a2ed98903648260760c44ec5b69ad94dbd45689ac WatchSource:0}: Error finding container 7846af4c430cbe18afc7856a2ed98903648260760c44ec5b69ad94dbd45689ac: Status 404 returned error can't find the container with id 7846af4c430cbe18afc7856a2ed98903648260760c44ec5b69ad94dbd45689ac
Oct 30 01:18:33.943779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:33.943752    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"]
Oct 30 01:18:34.281346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.281263    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj" event={"ID":"86d9557d-48f8-409a-9cbc-22b8a39739c3","Type":"ContainerStarted","Data":"7846af4c430cbe18afc7856a2ed98903648260760c44ec5b69ad94dbd45689ac"}
Oct 30 01:18:34.283211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.283180    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"
Oct 30 01:18:34.283380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.283211    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4" event={"ID":"b950c829-f560-490a-8265-f6ef547abfee","Type":"ContainerDied","Data":"3758b2c14d8400cf5cf7237a49866ec90fa0dc78a88adf41b7b019d37b58043b"}
Oct 30 01:18:34.283380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.283280    5104 scope.go:122] "RemoveContainer" containerID="a36f637b90fca3c6ac4585d4fbb892926915d19e915dfc07fb6591ce45392c85"
Oct 30 01:18:34.334497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.334442    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"]
Oct 30 01:18:34.345410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.345378    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e92dab-76bcb94f8b-hs4z4"]
Oct 30 01:18:34.618480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.618371    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02c7785-9248-485d-8d94-23290c4f6c0d" path="/var/lib/kubelet/pods/b02c7785-9248-485d-8d94-23290c4f6c0d/volumes"
Oct 30 01:18:34.619115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.619091    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b950c829-f560-490a-8265-f6ef547abfee" path="/var/lib/kubelet/pods/b950c829-f560-490a-8265-f6ef547abfee/volumes"
Oct 30 01:18:34.972686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.972634    5104 patch_prober.go:29] interesting pod/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:18:34.972959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:34.972927    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:18:35.116805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.116772    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:18:35.248358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.248325    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-envoy\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.248603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.248415    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69c9213-1688-410c-8e09-06ac2dc53702-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69c9213-1688-410c-8e09-06ac2dc53702-istiod-ca-cert\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.248603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.248562    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-data\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.248821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.248692    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a69c9213-1688-410c-8e09-06ac2dc53702-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69c9213-1688-410c-8e09-06ac2dc53702-istio-podinfo\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.248889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.248831    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-kube-api-access-l4482\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-kube-api-access-l4482\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.248889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.248854    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-socket\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.249164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.249131    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-socket" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:35.249286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.249197    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-credential-socket\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.249286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.249236    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-certs\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.249286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.249281    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-istio-token\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-istio-token\") pod \"a69c9213-1688-410c-8e09-06ac2dc53702\" (UID: \"a69c9213-1688-410c-8e09-06ac2dc53702\") "
Oct 30 01:18:35.250164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.249842    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69c9213-1688-410c-8e09-06ac2dc53702-istiod-ca-cert" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:35.250164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.249849    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-certs" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:35.250164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.250120    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-credential-socket" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:35.250409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.250338    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-data" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:35.250749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.250726    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69c9213-1688-410c-8e09-06ac2dc53702-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.250848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.250754    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.250848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.250769    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.250848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.250783    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.250848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.250796    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.251372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.251343    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-envoy" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:35.252119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.252093    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a69c9213-1688-410c-8e09-06ac2dc53702-istio-podinfo" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:18:35.252421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.252396    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-istio-token" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:35.252900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.252880    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-kube-api-access-l4482" pod "a69c9213-1688-410c-8e09-06ac2dc53702" (UID: "a69c9213-1688-410c-8e09-06ac2dc53702"). InnerVolumeSpecName "kube-api-access-l4482". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:35.300826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.300791    5104 generic.go:356] "Generic (PLEG): container finished" podID="a69c9213-1688-410c-8e09-06ac2dc53702" containerID="0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f" exitCode=0
Oct 30 01:18:35.301338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.300856    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"
Oct 30 01:18:35.301338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.300873    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" event={"ID":"a69c9213-1688-410c-8e09-06ac2dc53702","Type":"ContainerDied","Data":"0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f"}
Oct 30 01:18:35.301338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.300943    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh" event={"ID":"a69c9213-1688-410c-8e09-06ac2dc53702","Type":"ContainerDied","Data":"8714dab35b714e6774ff8ac03f7420fe180f9741ca69e3193c3e06ae445fee5c"}
Oct 30 01:18:35.301338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.300976    5104 scope.go:122] "RemoveContainer" containerID="0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f"
Oct 30 01:18:35.307181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.307130    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj" event={"ID":"86d9557d-48f8-409a-9cbc-22b8a39739c3","Type":"ContainerStarted","Data":"7d346b044524ab6db16e0ef2cbb5065a6f90963882a70c9db6cd60090181ddb1"}
Oct 30 01:18:35.321728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.321705    5104 scope.go:122] "RemoveContainer" containerID="0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2"
Oct 30 01:18:35.333860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.333833    5104 scope.go:122] "RemoveContainer" containerID="0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f"
Oct 30 01:18:35.334204 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:35.334183    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f\": container with ID starting with 0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f not found: ID does not exist" containerID="0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f"
Oct 30 01:18:35.334317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.334211    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f"} err="failed to get container status \"0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f\": rpc error: code = NotFound desc = could not find container \"0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f\": container with ID starting with 0d8aea3db1cf6f31b742d3c0114fa45ba7e0341606960682fdb6edc918ec043f not found: ID does not exist"
Oct 30 01:18:35.334317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.334227    5104 scope.go:122] "RemoveContainer" containerID="0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2"
Oct 30 01:18:35.334528 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:35.334511    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2\": container with ID starting with 0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2 not found: ID does not exist" containerID="0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2"
Oct 30 01:18:35.334573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.334534    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2"} err="failed to get container status \"0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2\": rpc error: code = NotFound desc = could not find container \"0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2\": container with ID starting with 0bef2f60e4b32a606e5620942dec9b07d9830188d06c304e375694f7755a76f2 not found: ID does not exist"
Oct 30 01:18:35.352339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.352299    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69c9213-1688-410c-8e09-06ac2dc53702-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.352339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.352342    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69c9213-1688-410c-8e09-06ac2dc53702-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.352580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.352356    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4482\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-kube-api-access-l4482\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.352580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.352372    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69c9213-1688-410c-8e09-06ac2dc53702-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:35.352580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.352451    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj" podStartSLOduration=2.3524393200000002 podStartE2EDuration="2.35243932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:35.351284367 +0000 UTC m=+2156.898974865" watchObservedRunningTime="2026-10-30 01:18:35.35243932 +0000 UTC m=+2156.900129816"
Oct 30 01:18:35.390764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.390725    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"]
Oct 30 01:18:35.396198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:35.396154    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-2pskh"]
Oct 30 01:18:36.324934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:36.324899    5104 generic.go:356] "Generic (PLEG): container finished" podID="86d9557d-48f8-409a-9cbc-22b8a39739c3" containerID="7d346b044524ab6db16e0ef2cbb5065a6f90963882a70c9db6cd60090181ddb1" exitCode=0
Oct 30 01:18:36.325370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:36.324992    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj" event={"ID":"86d9557d-48f8-409a-9cbc-22b8a39739c3","Type":"ContainerDied","Data":"7d346b044524ab6db16e0ef2cbb5065a6f90963882a70c9db6cd60090181ddb1"}
Oct 30 01:18:36.846799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:36.845404    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" path="/var/lib/kubelet/pods/a69c9213-1688-410c-8e09-06ac2dc53702/volumes"
Oct 30 01:18:36.924515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:36.922956    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"]
Oct 30 01:18:37.705480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.701805    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:37.887870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.887835    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-openshift-ca\") pod \"86d9557d-48f8-409a-9cbc-22b8a39739c3\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") "
Oct 30 01:18:37.888089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.888072    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-xks-ca\") pod \"86d9557d-48f8-409a-9cbc-22b8a39739c3\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") "
Oct 30 01:18:37.888223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.888209    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d9557d-48f8-409a-9cbc-22b8a39739c3-kube-api-access-z689r\" (UniqueName: \"kubernetes.io/projected/86d9557d-48f8-409a-9cbc-22b8a39739c3-kube-api-access-z689r\") pod \"86d9557d-48f8-409a-9cbc-22b8a39739c3\" (UID: \"86d9557d-48f8-409a-9cbc-22b8a39739c3\") "
Oct 30 01:18:37.889490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.889193    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-openshift-ca" pod "86d9557d-48f8-409a-9cbc-22b8a39739c3" (UID: "86d9557d-48f8-409a-9cbc-22b8a39739c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:37.914589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.914524    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-xks-ca" pod "86d9557d-48f8-409a-9cbc-22b8a39739c3" (UID: "86d9557d-48f8-409a-9cbc-22b8a39739c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:37.915008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.914960    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d9557d-48f8-409a-9cbc-22b8a39739c3-kube-api-access-z689r" pod "86d9557d-48f8-409a-9cbc-22b8a39739c3" (UID: "86d9557d-48f8-409a-9cbc-22b8a39739c3"). InnerVolumeSpecName "kube-api-access-z689r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:37.991286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.991248    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:37.991286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.991291    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z689r\" (UniqueName: \"kubernetes.io/projected/86d9557d-48f8-409a-9cbc-22b8a39739c3-kube-api-access-z689r\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:37.991531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:37.991308    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d9557d-48f8-409a-9cbc-22b8a39739c3-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.009488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.009434    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"]
Oct 30 01:18:38.197550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.196593    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.302864    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-credential-socket\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.302995    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-socket\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.303051    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-envoy\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.303096    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-token\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-token\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.303145    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-podinfo\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.303309    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-kube-api-access-9ck9f\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-kube-api-access-9ck9f\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.303443    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8de5c9d-68d2-4468-be09-b9eee683918d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de5c9d-68d2-4468-be09-b9eee683918d-istiod-ca-cert\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.303532    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-certs\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.304485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.303646    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-data\") pod \"c8de5c9d-68d2-4468-be09-b9eee683918d\" (UID: \"c8de5c9d-68d2-4468-be09-b9eee683918d\") "
Oct 30 01:18:38.315070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.315016    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-certs" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:38.315447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.315400    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8de5c9d-68d2-4468-be09-b9eee683918d-istiod-ca-cert" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:38.315447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.315491    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-socket" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:38.315816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.315793    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-data" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:38.330801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.323486    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-credential-socket" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:38.339785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.333874    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-kube-api-access-9ck9f" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "kube-api-access-9ck9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:38.350547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.350507    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-envoy" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:38.350547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.350518    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-podinfo" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:18:38.351684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.351650    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-token" pod "c8de5c9d-68d2-4468-be09-b9eee683918d" (UID: "c8de5c9d-68d2-4468-be09-b9eee683918d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:38.367377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.367338    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"
Oct 30 01:18:38.367377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.367341    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj" event={"ID":"86d9557d-48f8-409a-9cbc-22b8a39739c3","Type":"ContainerDied","Data":"7846af4c430cbe18afc7856a2ed98903648260760c44ec5b69ad94dbd45689ac"}
Oct 30 01:18:38.367663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.367418    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7846af4c430cbe18afc7856a2ed98903648260760c44ec5b69ad94dbd45689ac"
Oct 30 01:18:38.371097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.371062    5104 generic.go:356] "Generic (PLEG): container finished" podID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerID="d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6" exitCode=0
Oct 30 01:18:38.371348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.371322    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" event={"ID":"c8de5c9d-68d2-4468-be09-b9eee683918d","Type":"ContainerDied","Data":"d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6"}
Oct 30 01:18:38.371415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.371387    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr" event={"ID":"c8de5c9d-68d2-4468-be09-b9eee683918d","Type":"ContainerDied","Data":"87f5ef63b02c64499457f71a4b0dbb98947f37aa3076e2c8b5642dc173fb26dc"}
Oct 30 01:18:38.371492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.371419    5104 scope.go:122] "RemoveContainer" containerID="d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6"
Oct 30 01:18:38.371656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.371637    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"
Oct 30 01:18:38.397431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.397396    5104 scope.go:122] "RemoveContainer" containerID="44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3"
Oct 30 01:18:38.406901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.406866    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.406908    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.406928    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.406942    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.406956    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.406970    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ck9f\" (UniqueName: \"kubernetes.io/projected/c8de5c9d-68d2-4468-be09-b9eee683918d-kube-api-access-9ck9f\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.406988    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de5c9d-68d2-4468-be09-b9eee683918d-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.407002    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.407056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.407016    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de5c9d-68d2-4468-be09-b9eee683918d-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.415594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.415564    5104 scope.go:122] "RemoveContainer" containerID="d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6"
Oct 30 01:18:38.416021 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:38.415994    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6\": container with ID starting with d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6 not found: ID does not exist" containerID="d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6"
Oct 30 01:18:38.416106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.416031    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6"} err="failed to get container status \"d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6\": rpc error: code = NotFound desc = could not find container \"d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6\": container with ID starting with d2c266a1f3a72d7d450db135e3d149328350812f9d1bd55ad2d6e4840b249fc6 not found: ID does not exist"
Oct 30 01:18:38.416106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.416050    5104 scope.go:122] "RemoveContainer" containerID="44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3"
Oct 30 01:18:38.416428 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:38.416397    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3\": container with ID starting with 44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3 not found: ID does not exist" containerID="44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3"
Oct 30 01:18:38.416589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.416430    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3"} err="failed to get container status \"44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3\": rpc error: code = NotFound desc = could not find container \"44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3\": container with ID starting with 44f9527c9e4015c1aad13e28320123a0cfc3d8ae0bb0fde6aecb28eb7b47bab3 not found: ID does not exist"
Oct 30 01:18:38.497563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.492590    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"]
Oct 30 01:18:38.509933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.509894    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4dde48-openshift-default-7599b7cfcc-wg7cr"]
Oct 30 01:18:38.589717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:38.589680    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" path="/var/lib/kubelet/pods/c8de5c9d-68d2-4468-be09-b9eee683918d/volumes"
Oct 30 01:18:39.107042 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:39.107004    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:18:39.250907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:39.250871    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"]
Oct 30 01:18:39.251140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:39.251115    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy" containerID="cri-o://2cafa7121ee5459132b463806bad6c066b4eb80ee3e6051e10422fb0e9c47a8f" gracePeriod=30
Oct 30 01:18:41.923961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:41.923907    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" podUID="88842834-7005-4c5d-bade-eddca63b948b" containerName="payload-processing" containerID="cri-o://9da08a6905bff1aa05eded31b1c95d871660e522b9527575dd0390fb522428db" gracePeriod=25
Oct 30 01:18:42.674650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:42.674587    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5" podUID="b5802f4e-a81e-4630-98ce-b308b35f457b" containerName="main" containerID="cri-o://2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210" gracePeriod=45
Oct 30 01:18:43.001107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.001080    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:18:43.010626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.010585    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" podUID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" containerName="payload-pre-processing" containerID="cri-o://ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae" gracePeriod=25
Oct 30 01:18:43.037672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.037623    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg" podUID="9a27a7b6-042c-4a01-a8a7-79afa3b7f400" containerName="main" containerID="cri-o://2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54" gracePeriod=45
Oct 30 01:18:43.111273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111225    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-home\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-home\") pod \"b5802f4e-a81e-4630-98ce-b308b35f457b\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") "
Oct 30 01:18:43.111495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111300    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-model-cache\") pod \"b5802f4e-a81e-4630-98ce-b308b35f457b\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") "
Oct 30 01:18:43.111571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111504    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-tmp-dir\") pod \"b5802f4e-a81e-4630-98ce-b308b35f457b\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") "
Oct 30 01:18:43.111614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111555    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-home" pod "b5802f4e-a81e-4630-98ce-b308b35f457b" (UID: "b5802f4e-a81e-4630-98ce-b308b35f457b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.111614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111592    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5802f4e-a81e-4630-98ce-b308b35f457b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5802f4e-a81e-4630-98ce-b308b35f457b-tls-certs\") pod \"b5802f4e-a81e-4630-98ce-b308b35f457b\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") "
Oct 30 01:18:43.111730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111653    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-model-cache" pod "b5802f4e-a81e-4630-98ce-b308b35f457b" (UID: "b5802f4e-a81e-4630-98ce-b308b35f457b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.111796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111776    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-tmp-dir" pod "b5802f4e-a81e-4630-98ce-b308b35f457b" (UID: "b5802f4e-a81e-4630-98ce-b308b35f457b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.111867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111807    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-dshm\") pod \"b5802f4e-a81e-4630-98ce-b308b35f457b\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") "
Oct 30 01:18:43.111867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.111843    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5802f4e-a81e-4630-98ce-b308b35f457b-kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/b5802f4e-a81e-4630-98ce-b308b35f457b-kube-api-access-dqn8b\") pod \"b5802f4e-a81e-4630-98ce-b308b35f457b\" (UID: \"b5802f4e-a81e-4630-98ce-b308b35f457b\") "
Oct 30 01:18:43.112605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.112577    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.112605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.112607    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.112822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.112622    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.114741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.114717    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-dshm" pod "b5802f4e-a81e-4630-98ce-b308b35f457b" (UID: "b5802f4e-a81e-4630-98ce-b308b35f457b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.114858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.114815    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5802f4e-a81e-4630-98ce-b308b35f457b-kube-api-access-dqn8b" pod "b5802f4e-a81e-4630-98ce-b308b35f457b" (UID: "b5802f4e-a81e-4630-98ce-b308b35f457b"). InnerVolumeSpecName "kube-api-access-dqn8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:43.114962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.114936    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5802f4e-a81e-4630-98ce-b308b35f457b-tls-certs" pod "b5802f4e-a81e-4630-98ce-b308b35f457b" (UID: "b5802f4e-a81e-4630-98ce-b308b35f457b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:43.224495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.223917    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5802f4e-a81e-4630-98ce-b308b35f457b-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.224495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.223960    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5802f4e-a81e-4630-98ce-b308b35f457b-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.224495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.223976    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/b5802f4e-a81e-4630-98ce-b308b35f457b-kube-api-access-dqn8b\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.419632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.419599    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:18:43.426799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.426759    5104 generic.go:356] "Generic (PLEG): container finished" podID="9a27a7b6-042c-4a01-a8a7-79afa3b7f400" containerID="2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54" exitCode=0
Oct 30 01:18:43.427077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.427060    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"
Oct 30 01:18:43.427223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.427203    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg" event={"ID":"9a27a7b6-042c-4a01-a8a7-79afa3b7f400","Type":"ContainerDied","Data":"2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54"}
Oct 30 01:18:43.427287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.427245    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg" event={"ID":"9a27a7b6-042c-4a01-a8a7-79afa3b7f400","Type":"ContainerDied","Data":"a636181da5439fc2a4d454fa163533ec5b0aa063a5e7e34b60ae89c128dd9c8a"}
Oct 30 01:18:43.427287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.427268    5104 scope.go:122] "RemoveContainer" containerID="2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54"
Oct 30 01:18:43.434707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.429231    5104 generic.go:356] "Generic (PLEG): container finished" podID="b5802f4e-a81e-4630-98ce-b308b35f457b" containerID="2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210" exitCode=0
Oct 30 01:18:43.434707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.429281    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5" event={"ID":"b5802f4e-a81e-4630-98ce-b308b35f457b","Type":"ContainerDied","Data":"2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210"}
Oct 30 01:18:43.434707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.429289    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"
Oct 30 01:18:43.434707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.429313    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5" event={"ID":"b5802f4e-a81e-4630-98ce-b308b35f457b","Type":"ContainerDied","Data":"aa4c797a1d11d75d91491d85d3090e7fa9d36d955eca8d375b6b23bbe6cc98f7"}
Oct 30 01:18:43.441151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.441131    5104 scope.go:122] "RemoveContainer" containerID="2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54"
Oct 30 01:18:43.441494 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:43.441434    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54\": container with ID starting with 2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54 not found: ID does not exist" containerID="2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54"
Oct 30 01:18:43.441494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.441479    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54"} err="failed to get container status \"2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54\": rpc error: code = NotFound desc = could not find container \"2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54\": container with ID starting with 2853aa89139aefeb25748ce18eecbef723a0c809249f48e37c0e734811d76d54 not found: ID does not exist"
Oct 30 01:18:43.441494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.441497    5104 scope.go:122] "RemoveContainer" containerID="2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210"
Oct 30 01:18:43.453977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.453955    5104 scope.go:122] "RemoveContainer" containerID="2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210"
Oct 30 01:18:43.454315 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:43.454292    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210\": container with ID starting with 2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210 not found: ID does not exist" containerID="2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210"
Oct 30 01:18:43.454383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.454324    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210"} err="failed to get container status \"2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210\": rpc error: code = NotFound desc = could not find container \"2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210\": container with ID starting with 2f835bc9b13803f242edb3bf73c61bff74571d7a16cbd33bbfc969e89896f210 not found: ID does not exist"
Oct 30 01:18:43.514598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.514427    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"]
Oct 30 01:18:43.522722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.522687    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e69dee/test-model-e69dee-kserve-8594c4554c-2l6r5"]
Oct 30 01:18:43.528807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.528773    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tls-certs\") pod \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") "
Oct 30 01:18:43.528985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.528964    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-kube-api-access-4ss5v\" (UniqueName: \"kubernetes.io/projected/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-kube-api-access-4ss5v\") pod \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") "
Oct 30 01:18:43.529054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.529037    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-dshm\") pod \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") "
Oct 30 01:18:43.529195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.529173    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-model-cache\") pod \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") "
Oct 30 01:18:43.529300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.529262    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tmp-dir\") pod \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") "
Oct 30 01:18:43.529365 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.529327    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-home\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-home\") pod \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\" (UID: \"9a27a7b6-042c-4a01-a8a7-79afa3b7f400\") "
Oct 30 01:18:43.530368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.529739    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tmp-dir" pod "9a27a7b6-042c-4a01-a8a7-79afa3b7f400" (UID: "9a27a7b6-042c-4a01-a8a7-79afa3b7f400"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.530368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.529978    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-home" pod "9a27a7b6-042c-4a01-a8a7-79afa3b7f400" (UID: "9a27a7b6-042c-4a01-a8a7-79afa3b7f400"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.530368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.530306    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-model-cache" pod "9a27a7b6-042c-4a01-a8a7-79afa3b7f400" (UID: "9a27a7b6-042c-4a01-a8a7-79afa3b7f400"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.530607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.530368    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.530607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.530389    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.531964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.531938    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-kube-api-access-4ss5v" pod "9a27a7b6-042c-4a01-a8a7-79afa3b7f400" (UID: "9a27a7b6-042c-4a01-a8a7-79afa3b7f400"). InnerVolumeSpecName "kube-api-access-4ss5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:43.532096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.532077    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-dshm" pod "9a27a7b6-042c-4a01-a8a7-79afa3b7f400" (UID: "9a27a7b6-042c-4a01-a8a7-79afa3b7f400"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:43.532297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.532271    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tls-certs" pod "9a27a7b6-042c-4a01-a8a7-79afa3b7f400" (UID: "9a27a7b6-042c-4a01-a8a7-79afa3b7f400"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:43.632415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.632374    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ss5v\" (UniqueName: \"kubernetes.io/projected/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-kube-api-access-4ss5v\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.632415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.632415    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.632696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.632432    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.632696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.632446    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a27a7b6-042c-4a01-a8a7-79afa3b7f400-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.720958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.720925    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"]
Oct 30 01:18:43.753745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.753708    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"]
Oct 30 01:18:43.848598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.837726    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"]
Oct 30 01:18:43.859177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:43.859136    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-50d078/test-model-50d078-kserve-568bcf4f8c-h49zg"]
Oct 30 01:18:44.673173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.673137    5104 generic.go:356] "Generic (PLEG): container finished" podID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerID="2cafa7121ee5459132b463806bad6c066b4eb80ee3e6051e10422fb0e9c47a8f" exitCode=0
Oct 30 01:18:44.675672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.675579    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a27a7b6-042c-4a01-a8a7-79afa3b7f400" path="/var/lib/kubelet/pods/9a27a7b6-042c-4a01-a8a7-79afa3b7f400/volumes"
Oct 30 01:18:44.676285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.676264    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5802f4e-a81e-4630-98ce-b308b35f457b" path="/var/lib/kubelet/pods/b5802f4e-a81e-4630-98ce-b308b35f457b/volumes"
Oct 30 01:18:44.676755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.676728    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" event={"ID":"d03188ad-e1df-4bdf-bcac-b9d46a5e704b","Type":"ContainerDied","Data":"2cafa7121ee5459132b463806bad6c066b4eb80ee3e6051e10422fb0e9c47a8f"}
Oct 30 01:18:44.676848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.676797    5104 scope.go:122] "RemoveContainer" containerID="0a0bd83d509ee3c6cf4f080b5691f1eb6f9be5bdf00bbf00fd59b88ec95eaf25"
Oct 30 01:18:44.867003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.866976    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 01:18:44.941536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.941423    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-kube-api-access-r8p28\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-kube-api-access-r8p28\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.941775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.941562    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-podinfo\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.941775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.941647    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-token\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-token\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.941775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.941679    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-certs\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.942010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.941856    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-socket\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.942010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.941972    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istiod-ca-cert\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.942155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942041    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-data\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.942155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942070    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-envoy\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.942155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942105    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-credential-socket\") pod \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\" (UID: \"d03188ad-e1df-4bdf-bcac-b9d46a5e704b\") "
Oct 30 01:18:44.942603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942574    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-data" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:44.942818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942792    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-credential-socket" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:44.942893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942582    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istiod-ca-cert" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:44.942893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942852    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-certs" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:44.943068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.942993    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-socket" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:44.943146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.943129    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:44.943207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.943148    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:44.943207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.943163    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:44.943207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.943176    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:44.943207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.943190    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:44.944557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.944521    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-kube-api-access-r8p28" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "kube-api-access-r8p28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:44.945230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.945179    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-envoy" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:44.946807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.946771    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-token" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:44.949769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:44.949743    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-podinfo" pod "d03188ad-e1df-4bdf-bcac-b9d46a5e704b" (UID: "d03188ad-e1df-4bdf-bcac-b9d46a5e704b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:18:45.048880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.047860    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:45.048880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.047905    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8p28\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-kube-api-access-r8p28\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:45.048880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.047923    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:45.048880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.047936    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03188ad-e1df-4bdf-bcac-b9d46a5e704b-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:45.686583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.686550    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"
Oct 30 01:18:45.687056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.686550    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm" event={"ID":"d03188ad-e1df-4bdf-bcac-b9d46a5e704b","Type":"ContainerDied","Data":"219096a11fb0ff4c61e43afd73606b1d2e3186dc066b816edf3acb0a287f08c8"}
Oct 30 01:18:45.687056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.686724    5104 scope.go:122] "RemoveContainer" containerID="2cafa7121ee5459132b463806bad6c066b4eb80ee3e6051e10422fb0e9c47a8f"
Oct 30 01:18:45.739992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.739950    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"]
Oct 30 01:18:45.750391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:45.750356    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-ws8gm"]
Oct 30 01:18:46.457539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:46.457489    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" podUID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:46.582139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:46.582092    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" path="/var/lib/kubelet/pods/d03188ad-e1df-4bdf-bcac-b9d46a5e704b/volumes"
Oct 30 01:18:46.888869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:46.888771    5104 scope.go:122] "RemoveContainer" containerID="5edc3708cb59cc62fe7766eafc1754ccc7eed7be37fe3034d0b5c541d6236801"
Oct 30 01:18:48.560217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:48.560178    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" podUID="88842834-7005-4c5d-bade-eddca63b948b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:48.721862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:48.721793    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q" podUID="de1e8764-ed7d-4878-9b31-d2aae18204fd" containerName="payload-processing" containerID="cri-o://0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb" gracePeriod=25
Oct 30 01:18:48.754603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:48.754548    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" podUID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" containerName="payload-pre-processing" containerID="cri-o://4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4" gracePeriod=25
Oct 30 01:18:49.046198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.046169    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:18:49.055386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.055362    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1e8764-ed7d-4878-9b31-d2aae18204fd-kube-api-access-6zg7q\" (UniqueName: \"kubernetes.io/projected/de1e8764-ed7d-4878-9b31-d2aae18204fd-kube-api-access-6zg7q\") pod \"de1e8764-ed7d-4878-9b31-d2aae18204fd\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") "
Oct 30 01:18:49.055545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.055529    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1e8764-ed7d-4878-9b31-d2aae18204fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1e8764-ed7d-4878-9b31-d2aae18204fd-plugins-config-volume\") pod \"de1e8764-ed7d-4878-9b31-d2aae18204fd\" (UID: \"de1e8764-ed7d-4878-9b31-d2aae18204fd\") "
Oct 30 01:18:49.055929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.055906    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1e8764-ed7d-4878-9b31-d2aae18204fd-plugins-config-volume" pod "de1e8764-ed7d-4878-9b31-d2aae18204fd" (UID: "de1e8764-ed7d-4878-9b31-d2aae18204fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:49.056057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.056043    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1e8764-ed7d-4878-9b31-d2aae18204fd-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:49.058057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.058030    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1e8764-ed7d-4878-9b31-d2aae18204fd-kube-api-access-6zg7q" pod "de1e8764-ed7d-4878-9b31-d2aae18204fd" (UID: "de1e8764-ed7d-4878-9b31-d2aae18204fd"). InnerVolumeSpecName "kube-api-access-6zg7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:49.157411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.157319    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zg7q\" (UniqueName: \"kubernetes.io/projected/de1e8764-ed7d-4878-9b31-d2aae18204fd-kube-api-access-6zg7q\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:49.729143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.729107    5104 generic.go:356] "Generic (PLEG): container finished" podID="de1e8764-ed7d-4878-9b31-d2aae18204fd" containerID="0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb" exitCode=0
Oct 30 01:18:49.729689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.729162    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"
Oct 30 01:18:49.729689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.729193    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q" event={"ID":"de1e8764-ed7d-4878-9b31-d2aae18204fd","Type":"ContainerDied","Data":"0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb"}
Oct 30 01:18:49.729689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.729253    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q" event={"ID":"de1e8764-ed7d-4878-9b31-d2aae18204fd","Type":"ContainerDied","Data":"d2f7ff56a411c087ae3855322b5de59f50447f865910c69a2979f64cd945956e"}
Oct 30 01:18:49.729689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.729275    5104 scope.go:122] "RemoveContainer" containerID="0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb"
Oct 30 01:18:49.742226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.742204    5104 scope.go:122] "RemoveContainer" containerID="0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb"
Oct 30 01:18:49.742616 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:18:49.742581    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb\": container with ID starting with 0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb not found: ID does not exist" containerID="0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb"
Oct 30 01:18:49.742722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.742621    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb"} err="failed to get container status \"0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb\": rpc error: code = NotFound desc = could not find container \"0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb\": container with ID starting with 0c7c402e6899a7384d476579d4a95b0e9b9c0800f49cb7fc015b8330c5b732bb not found: ID does not exist"
Oct 30 01:18:49.781544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.781508    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"]
Oct 30 01:18:49.792377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:49.792346    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e896-6fbfdbbf68-cr55q"]
Oct 30 01:18:50.583818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:50.583779    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1e8764-ed7d-4878-9b31-d2aae18204fd" path="/var/lib/kubelet/pods/de1e8764-ed7d-4878-9b31-d2aae18204fd/volumes"
Oct 30 01:18:55.785161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:55.783314    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"]
Oct 30 01:18:56.000817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.000776    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"]
Oct 30 01:18:56.002415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.002347    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9a27a7b6-042c-4a01-a8a7-79afa3b7f400" containerName="main"
Oct 30 01:18:56.002697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.002666    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a27a7b6-042c-4a01-a8a7-79afa3b7f400" containerName="main"
Oct 30 01:18:56.002826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.002815    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy"
Oct 30 01:18:56.002911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.002900    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy"
Oct 30 01:18:56.002998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.002985    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy"
Oct 30 01:18:56.003078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003069    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy"
Oct 30 01:18:56.003190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003181    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy"
Oct 30 01:18:56.003269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003259    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy"
Oct 30 01:18:56.003358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003349    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy"
Oct 30 01:18:56.003428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003419    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy"
Oct 30 01:18:56.003573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003558    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy"
Oct 30 01:18:56.003665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003654    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy"
Oct 30 01:18:56.003750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003740    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b950c829-f560-490a-8265-f6ef547abfee" containerName="payload-processing"
Oct 30 01:18:56.003834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003824    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b950c829-f560-490a-8265-f6ef547abfee" containerName="payload-processing"
Oct 30 01:18:56.003925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003915    5104 cpu_manager.go:414] "containerMap: removing container" podUID="de1e8764-ed7d-4878-9b31-d2aae18204fd" containerName="payload-processing"
Oct 30 01:18:56.004003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.003993    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1e8764-ed7d-4878-9b31-d2aae18204fd" containerName="payload-processing"
Oct 30 01:18:56.004099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004090    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b5802f4e-a81e-4630-98ce-b308b35f457b" containerName="main"
Oct 30 01:18:56.004180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004170    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5802f4e-a81e-4630-98ce-b308b35f457b" containerName="main"
Oct 30 01:18:56.004270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004261    5104 cpu_manager.go:414] "containerMap: removing container" podUID="86d9557d-48f8-409a-9cbc-22b8a39739c3" containerName="revoke-keys"
Oct 30 01:18:56.004346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004337    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d9557d-48f8-409a-9cbc-22b8a39739c3" containerName="revoke-keys"
Oct 30 01:18:56.004694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004677    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1e8764-ed7d-4878-9b31-d2aae18204fd" containerName="payload-processing"
Oct 30 01:18:56.004802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004792    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5802f4e-a81e-4630-98ce-b308b35f457b" containerName="main"
Oct 30 01:18:56.004897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004887    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d9557d-48f8-409a-9cbc-22b8a39739c3" containerName="revoke-keys"
Oct 30 01:18:56.004981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.004973    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a27a7b6-042c-4a01-a8a7-79afa3b7f400" containerName="main"
Oct 30 01:18:56.005061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.005051    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy"
Oct 30 01:18:56.005144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.005134    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy"
Oct 30 01:18:56.005233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.005223    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy"
Oct 30 01:18:56.005318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.005309    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03188ad-e1df-4bdf-bcac-b9d46a5e704b" containerName="istio-proxy"
Oct 30 01:18:56.005398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.005389    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b950c829-f560-490a-8265-f6ef547abfee" containerName="payload-processing"
Oct 30 01:18:56.072274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.072190    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"]
Oct 30 01:18:56.072672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.072646    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.086829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.086797    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:18:56.087192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.087169    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:18:56.088137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.087313    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:18:56.088137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.087400    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:18:56.213340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.213300    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.213565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.213525    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.213646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.213630    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8s2\" (UniqueName: \"kubernetes.io/projected/8078d2de-d859-480b-986e-91d258a33dc4-kube-api-access-7j8s2\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.324279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.324159    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.324488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.324341    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8s2\" (UniqueName: \"kubernetes.io/projected/8078d2de-d859-480b-986e-91d258a33dc4-kube-api-access-7j8s2\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.324565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.324517    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.325255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.325229    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.326824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.326792    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.342838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.342809    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8s2\" (UniqueName: \"kubernetes.io/projected/8078d2de-d859-480b-986e-91d258a33dc4-kube-api-access-7j8s2\") pod \"maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.415356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.415319    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:56.459167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.459121    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" podUID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:56.727067 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:18:56.727036    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8078d2de_d859_480b_986e_91d258a33dc4.slice/crio-f2fe95466711f5547f651eb893493191d1572ae6b94377ed137c579aa799c59f WatchSource:0}: Error finding container f2fe95466711f5547f651eb893493191d1572ae6b94377ed137c579aa799c59f: Status 404 returned error can't find the container with id f2fe95466711f5547f651eb893493191d1572ae6b94377ed137c579aa799c59f
Oct 30 01:18:56.727349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.727324    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"]
Oct 30 01:18:56.822610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:56.822571    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp" event={"ID":"8078d2de-d859-480b-986e-91d258a33dc4","Type":"ContainerStarted","Data":"f2fe95466711f5547f651eb893493191d1572ae6b94377ed137c579aa799c59f"}
Oct 30 01:18:57.253910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:57.253818    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" podUID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:57.834602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:57.834564    5104 generic.go:356] "Generic (PLEG): container finished" podID="8078d2de-d859-480b-986e-91d258a33dc4" containerID="2e47d2c9547695cc2c8da84f2da86e8f41b6ab601e54609261acbf776e98d90f" exitCode=0
Oct 30 01:18:57.835018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:57.834645    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp" event={"ID":"8078d2de-d859-480b-986e-91d258a33dc4","Type":"ContainerDied","Data":"2e47d2c9547695cc2c8da84f2da86e8f41b6ab601e54609261acbf776e98d90f"}
Oct 30 01:18:58.559538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:58.559497    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" podUID="88842834-7005-4c5d-bade-eddca63b948b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:59.058911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.058879    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:59.177575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.177537    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-xks-ca\") pod \"8078d2de-d859-480b-986e-91d258a33dc4\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") "
Oct 30 01:18:59.177789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.177655    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-openshift-ca\") pod \"8078d2de-d859-480b-986e-91d258a33dc4\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") "
Oct 30 01:18:59.177852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.177824    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8078d2de-d859-480b-986e-91d258a33dc4-kube-api-access-7j8s2\" (UniqueName: \"kubernetes.io/projected/8078d2de-d859-480b-986e-91d258a33dc4-kube-api-access-7j8s2\") pod \"8078d2de-d859-480b-986e-91d258a33dc4\" (UID: \"8078d2de-d859-480b-986e-91d258a33dc4\") "
Oct 30 01:18:59.178042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.178005    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-openshift-ca" pod "8078d2de-d859-480b-986e-91d258a33dc4" (UID: "8078d2de-d859-480b-986e-91d258a33dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:59.178626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.178605    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:59.179919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.179896    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-xks-ca" pod "8078d2de-d859-480b-986e-91d258a33dc4" (UID: "8078d2de-d859-480b-986e-91d258a33dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:59.180325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.180307    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8078d2de-d859-480b-986e-91d258a33dc4-kube-api-access-7j8s2" pod "8078d2de-d859-480b-986e-91d258a33dc4" (UID: "8078d2de-d859-480b-986e-91d258a33dc4"). InnerVolumeSpecName "kube-api-access-7j8s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:59.280633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.280595    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8s2\" (UniqueName: \"kubernetes.io/projected/8078d2de-d859-480b-986e-91d258a33dc4-kube-api-access-7j8s2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:59.280633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.280633    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8078d2de-d859-480b-986e-91d258a33dc4-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:18:59.867821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.867790    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"
Oct 30 01:18:59.868023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.867827    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp" event={"ID":"8078d2de-d859-480b-986e-91d258a33dc4","Type":"ContainerDied","Data":"f2fe95466711f5547f651eb893493191d1572ae6b94377ed137c579aa799c59f"}
Oct 30 01:18:59.868023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:18:59.867885    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2fe95466711f5547f651eb893493191d1572ae6b94377ed137c579aa799c59f"
Oct 30 01:19:00.845579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:00.845543    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"]
Oct 30 01:19:00.856627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:00.856592    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82e9bf-kgmmn"]
Oct 30 01:19:03.257927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.257888    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5f78e4-398b-4d80-80a1-14377945ee5a" path="/var/lib/kubelet/pods/9b5f78e4-398b-4d80-80a1-14377945ee5a/volumes"
Oct 30 01:19:03.277281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.276970    5104 generic.go:356] "Generic (PLEG): container finished" podID="88842834-7005-4c5d-bade-eddca63b948b" containerID="9da08a6905bff1aa05eded31b1c95d871660e522b9527575dd0390fb522428db" exitCode=0
Oct 30 01:19:03.277281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.277111    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" event={"ID":"88842834-7005-4c5d-bade-eddca63b948b","Type":"ContainerDied","Data":"9da08a6905bff1aa05eded31b1c95d871660e522b9527575dd0390fb522428db"}
Oct 30 01:19:03.470442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.470396    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:19:03.577567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.577524    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88842834-7005-4c5d-bade-eddca63b948b-kube-api-access-4x7wl\" (UniqueName: \"kubernetes.io/projected/88842834-7005-4c5d-bade-eddca63b948b-kube-api-access-4x7wl\") pod \"88842834-7005-4c5d-bade-eddca63b948b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") "
Oct 30 01:19:03.578072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.578042    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88842834-7005-4c5d-bade-eddca63b948b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88842834-7005-4c5d-bade-eddca63b948b-plugins-config-volume\") pod \"88842834-7005-4c5d-bade-eddca63b948b\" (UID: \"88842834-7005-4c5d-bade-eddca63b948b\") "
Oct 30 01:19:03.579353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.579326    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88842834-7005-4c5d-bade-eddca63b948b-plugins-config-volume" pod "88842834-7005-4c5d-bade-eddca63b948b" (UID: "88842834-7005-4c5d-bade-eddca63b948b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:03.582998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.582958    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88842834-7005-4c5d-bade-eddca63b948b-kube-api-access-4x7wl" pod "88842834-7005-4c5d-bade-eddca63b948b" (UID: "88842834-7005-4c5d-bade-eddca63b948b"). InnerVolumeSpecName "kube-api-access-4x7wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:03.616262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.616235    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:19:03.679792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.679745    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-kube-api-access-tjknf\" (UniqueName: \"kubernetes.io/projected/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-kube-api-access-tjknf\") pod \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") "
Oct 30 01:19:03.679982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.679953    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-plugins-config-volume\") pod \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\" (UID: \"d51768a8-3a1a-47be-8ad9-27eca95f9b9e\") "
Oct 30 01:19:03.680336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.680308    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-plugins-config-volume" pod "d51768a8-3a1a-47be-8ad9-27eca95f9b9e" (UID: "d51768a8-3a1a-47be-8ad9-27eca95f9b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:03.680857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.680833    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88842834-7005-4c5d-bade-eddca63b948b-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:03.680930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.680865    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7wl\" (UniqueName: \"kubernetes.io/projected/88842834-7005-4c5d-bade-eddca63b948b-kube-api-access-4x7wl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:03.680930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.680880    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:03.682018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.681993    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-kube-api-access-tjknf" pod "d51768a8-3a1a-47be-8ad9-27eca95f9b9e" (UID: "d51768a8-3a1a-47be-8ad9-27eca95f9b9e"). InnerVolumeSpecName "kube-api-access-tjknf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:03.782336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:03.782285    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjknf\" (UniqueName: \"kubernetes.io/projected/d51768a8-3a1a-47be-8ad9-27eca95f9b9e-kube-api-access-tjknf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.293312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.293215    5104 generic.go:356] "Generic (PLEG): container finished" podID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" containerID="ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae" exitCode=0
Oct 30 01:19:04.293312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.293274    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"
Oct 30 01:19:04.293996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.293306    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" event={"ID":"d51768a8-3a1a-47be-8ad9-27eca95f9b9e","Type":"ContainerDied","Data":"ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae"}
Oct 30 01:19:04.293996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.293369    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht" event={"ID":"d51768a8-3a1a-47be-8ad9-27eca95f9b9e","Type":"ContainerDied","Data":"d4d5f8cfddb1abf3060247ab3a8ca943fe33ef81ab27d85a90cc38d8c86e2c87"}
Oct 30 01:19:04.293996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.293397    5104 scope.go:122] "RemoveContainer" containerID="ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae"
Oct 30 01:19:04.295548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.295527    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"
Oct 30 01:19:04.295692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.295545    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b" event={"ID":"88842834-7005-4c5d-bade-eddca63b948b","Type":"ContainerDied","Data":"7a5d2d229e9935ef80a3b8011bea21223cb7b11efe097c084b157233e7675218"}
Oct 30 01:19:04.307849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.307825    5104 scope.go:122] "RemoveContainer" containerID="ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae"
Oct 30 01:19:04.308903 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:04.308243    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae\": container with ID starting with ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae not found: ID does not exist" containerID="ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae"
Oct 30 01:19:04.308903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.308292    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae"} err="failed to get container status \"ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae\": rpc error: code = NotFound desc = could not find container \"ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae\": container with ID starting with ee93ff63fdb54e047daa3b72ab211fd9230b4d4f026178a19530e0c346fc11ae not found: ID does not exist"
Oct 30 01:19:04.308903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.308311    5104 scope.go:122] "RemoveContainer" containerID="9da08a6905bff1aa05eded31b1c95d871660e522b9527575dd0390fb522428db"
Oct 30 01:19:04.360110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.360079    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"]
Oct 30 01:19:04.370605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.370575    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-56d1d9-59985c8c84-kj79b"]
Oct 30 01:19:04.410405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.410371    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"]
Oct 30 01:19:04.419150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.419120    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82e9bf-75b57bb5cf-qjsht"]
Oct 30 01:19:04.630561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.628313    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88842834-7005-4c5d-bade-eddca63b948b" path="/var/lib/kubelet/pods/88842834-7005-4c5d-bade-eddca63b948b/volumes"
Oct 30 01:19:04.630561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:04.629070    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" path="/var/lib/kubelet/pods/d51768a8-3a1a-47be-8ad9-27eca95f9b9e/volumes"
Oct 30 01:19:05.129752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:05.129715    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"]
Oct 30 01:19:05.140595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:05.140567    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-56d1d9-dkvmx"]
Oct 30 01:19:06.444314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:06.444275    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"]
Oct 30 01:19:06.498906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:06.498872    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"]
Oct 30 01:19:06.583262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:06.583218    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca4b507-869b-400c-9fb4-b9c26f5ec4dc" path="/var/lib/kubelet/pods/9ca4b507-869b-400c-9fb4-b9c26f5ec4dc/volumes"
Oct 30 01:19:07.253324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:07.253287    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" podUID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:08.658001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:08.656491    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"]
Oct 30 01:19:08.658001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:08.657218    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" containerID="cri-o://2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438" gracePeriod=30
Oct 30 01:19:09.136932 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:09.136885    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:19:09.191914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.191884    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:19:09.377494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.377398    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-kube-api-access-5zbsb\" (UniqueName: \"kubernetes.io/projected/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-kube-api-access-5zbsb\") pod \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") "
Oct 30 01:19:09.377682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.377524    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-plugins-config-volume\") pod \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\" (UID: \"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9\") "
Oct 30 01:19:09.377982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.377939    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-plugins-config-volume" pod "cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" (UID: "cc4de0ed-a83f-4e33-9c10-74ab1d125ac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:09.378275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.378258    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.379849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.379820    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-kube-api-access-5zbsb" pod "cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" (UID: "cc4de0ed-a83f-4e33-9c10-74ab1d125ac9"). InnerVolumeSpecName "kube-api-access-5zbsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:09.404325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.404290    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"]
Oct 30 01:19:09.405782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.405557    5104 generic.go:356] "Generic (PLEG): container finished" podID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" containerID="4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4" exitCode=0
Oct 30 01:19:09.409031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.407413    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" event={"ID":"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9","Type":"ContainerDied","Data":"4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4"}
Oct 30 01:19:09.409031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.407570    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg" event={"ID":"cc4de0ed-a83f-4e33-9c10-74ab1d125ac9","Type":"ContainerDied","Data":"0a4632880437be73bfe917b401b6c7833dde27459ca39302cd241c746c502581"}
Oct 30 01:19:09.409031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.407604    5104 scope.go:122] "RemoveContainer" containerID="4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4"
Oct 30 01:19:09.409031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.407867    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"
Oct 30 01:19:09.427485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.411871    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50d078-9pdpp"]
Oct 30 01:19:09.463245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.463214    5104 scope.go:122] "RemoveContainer" containerID="4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4"
Oct 30 01:19:09.463707 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:09.463674    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4\": container with ID starting with 4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4 not found: ID does not exist" containerID="4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4"
Oct 30 01:19:09.463831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.463727    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4"} err="failed to get container status \"4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4\": rpc error: code = NotFound desc = could not find container \"4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4\": container with ID starting with 4b5b09ef2ccdecf629d53ac45c8b49deb1ec48dfd0b6171c09159512c28dc7e4 not found: ID does not exist"
Oct 30 01:19:09.474813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.474782    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"]
Oct 30 01:19:09.481058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.481009    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zbsb\" (UniqueName: \"kubernetes.io/projected/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9-kube-api-access-5zbsb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:09.483338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.483310    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"]
Oct 30 01:19:09.484570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484546    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" containerName="payload-pre-processing"
Oct 30 01:19:09.484669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484573    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" containerName="payload-pre-processing"
Oct 30 01:19:09.484669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484609    5104 cpu_manager.go:414] "containerMap: removing container" podUID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" containerName="payload-pre-processing"
Oct 30 01:19:09.484669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484617    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" containerName="payload-pre-processing"
Oct 30 01:19:09.484669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484633    5104 cpu_manager.go:414] "containerMap: removing container" podUID="88842834-7005-4c5d-bade-eddca63b948b" containerName="payload-processing"
Oct 30 01:19:09.484669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484641    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88842834-7005-4c5d-bade-eddca63b948b" containerName="payload-processing"
Oct 30 01:19:09.484669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484651    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8078d2de-d859-480b-986e-91d258a33dc4" containerName="revoke-keys"
Oct 30 01:19:09.484669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484658    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8078d2de-d859-480b-986e-91d258a33dc4" containerName="revoke-keys"
Oct 30 01:19:09.484908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484684    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy"
Oct 30 01:19:09.484908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484692    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy"
Oct 30 01:19:09.484908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484888    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="88842834-7005-4c5d-bade-eddca63b948b" containerName="payload-processing"
Oct 30 01:19:09.484908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484899    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8078d2de-d859-480b-986e-91d258a33dc4" containerName="revoke-keys"
Oct 30 01:19:09.485040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484913    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8de5c9d-68d2-4468-be09-b9eee683918d" containerName="istio-proxy"
Oct 30 01:19:09.485040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484923    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69c9213-1688-410c-8e09-06ac2dc53702" containerName="istio-proxy"
Oct 30 01:19:09.485040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484946    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d51768a8-3a1a-47be-8ad9-27eca95f9b9e" containerName="payload-pre-processing"
Oct 30 01:19:09.485040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.484960    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" containerName="payload-pre-processing"
Oct 30 01:19:09.495305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.495280    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e896-68848c7795-v8xfg"]
Oct 30 01:19:09.495417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.495385    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.504989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.504951    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"]
Oct 30 01:19:09.510105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.509975    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:19:09.510227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.510203    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:19:09.510315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.510301    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:19:09.510371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.510359    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:09.684019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.683982    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrcg\" (UniqueName: \"kubernetes.io/projected/9f40dd84-6d57-45f8-80c7-ada42b04e41c-kube-api-access-zkrcg\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.684414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.684124    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.684414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.684350    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.788484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.787557    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.788484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.787857    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.788484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.788082    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrcg\" (UniqueName: \"kubernetes.io/projected/9f40dd84-6d57-45f8-80c7-ada42b04e41c-kube-api-access-zkrcg\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.789830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.789791    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.824506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.819415    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrcg\" (UniqueName: \"kubernetes.io/projected/9f40dd84-6d57-45f8-80c7-ada42b04e41c-kube-api-access-zkrcg\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:09.824506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:09.819876    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:10.115492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:10.115392    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:10.420800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:10.420771    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"]
Oct 30 01:19:10.421610 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:10.421583    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f40dd84_6d57_45f8_80c7_ada42b04e41c.slice/crio-cc6cc17bb78c7f0f6bf11345ad0e4702ac00f2448248d34467b4d3307637604e WatchSource:0}: Error finding container cc6cc17bb78c7f0f6bf11345ad0e4702ac00f2448248d34467b4d3307637604e: Status 404 returned error can't find the container with id cc6cc17bb78c7f0f6bf11345ad0e4702ac00f2448248d34467b4d3307637604e
Oct 30 01:19:10.594546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:10.593258    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8078d2de-d859-480b-986e-91d258a33dc4" path="/var/lib/kubelet/pods/8078d2de-d859-480b-986e-91d258a33dc4/volumes"
Oct 30 01:19:10.594546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:10.593998    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4de0ed-a83f-4e33-9c10-74ab1d125ac9" path="/var/lib/kubelet/pods/cc4de0ed-a83f-4e33-9c10-74ab1d125ac9/volumes"
Oct 30 01:19:10.867482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:10.864141    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg" podUID="e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" containerName="main" containerID="cri-o://38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8" gracePeriod=45
Oct 30 01:19:11.331482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.329696    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:19:11.445374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.442783    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"]
Oct 30 01:19:11.449904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.449865    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tmp-dir\") pod \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") "
Oct 30 01:19:11.450081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.449916    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-model-cache\") pod \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") "
Oct 30 01:19:11.450081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.450000    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-dshm\") pod \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") "
Oct 30 01:19:11.450214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.450130    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tls-certs\") pod \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") "
Oct 30 01:19:11.450214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.450187    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-home\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-home\") pod \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") "
Oct 30 01:19:11.450329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.450228    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-kube-api-access-7nvs2\" (UniqueName: \"kubernetes.io/projected/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-kube-api-access-7nvs2\") pod \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\" (UID: \"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767\") "
Oct 30 01:19:11.451442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.451410    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz" podUID="ab79d576-7543-4573-bfca-2cfa6a6000d6" containerName="payload-processing" containerID="cri-o://c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1" gracePeriod=25
Oct 30 01:19:11.452950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.452675    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-model-cache" pod "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" (UID: "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:11.452950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.452757    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tmp-dir" pod "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" (UID: "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:11.452950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.452811    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e896-zp4rj"]
Oct 30 01:19:11.452950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.452904    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-home" pod "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" (UID: "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:11.457675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.456799    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-dshm" pod "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" (UID: "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:11.468898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.466647    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tls-certs" pod "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" (UID: "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:11.468898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.467246    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-kube-api-access-7nvs2" pod "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" (UID: "e0d7b9ba-12aa-4de5-b9d2-edba23bcf767"). InnerVolumeSpecName "kube-api-access-7nvs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:11.519672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.508726    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" podUID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" containerName="payload-pre-processing" containerID="cri-o://6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335" gracePeriod=25
Oct 30 01:19:11.572880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.572747    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:11.645480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.620146    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:11.702939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.702852    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:11.703177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.703159    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:11.703299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.703287    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:11.705540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.703393    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nvs2\" (UniqueName: \"kubernetes.io/projected/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767-kube-api-access-7nvs2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:11.705540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.701306    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"
Oct 30 01:19:11.716508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.712445    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs" podStartSLOduration=2.712422845 podStartE2EDuration="2.712422845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:11.674739474 +0000 UTC m=+2193.222429973" watchObservedRunningTime="2026-10-30 01:19:11.712422845 +0000 UTC m=+2193.260113342"
Oct 30 01:19:11.719606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.610372    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs" event={"ID":"9f40dd84-6d57-45f8-80c7-ada42b04e41c","Type":"ContainerStarted","Data":"972d54775caaae2de591865a0af937cd9d67fe8d56a1b69fb2831d8e3a7e9cde"}
Oct 30 01:19:11.719910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.719884    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs" event={"ID":"9f40dd84-6d57-45f8-80c7-ada42b04e41c","Type":"ContainerStarted","Data":"cc6cc17bb78c7f0f6bf11345ad0e4702ac00f2448248d34467b4d3307637604e"}
Oct 30 01:19:11.720034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.720023    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg" event={"ID":"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767","Type":"ContainerDied","Data":"38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8"}
Oct 30 01:19:11.720114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.720106    5104 scope.go:122] "RemoveContainer" containerID="38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8"
Oct 30 01:19:11.720289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.701249    5104 generic.go:356] "Generic (PLEG): container finished" podID="e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" containerID="38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8" exitCode=0
Oct 30 01:19:11.720492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.720476    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg" event={"ID":"e0d7b9ba-12aa-4de5-b9d2-edba23bcf767","Type":"ContainerDied","Data":"054bbac424fda5019388df3522af149fed87c206fc844af10ee82db49e7fdf96"}
Oct 30 01:19:11.743829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.743769    5104 scope.go:122] "RemoveContainer" containerID="38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8"
Oct 30 01:19:11.744230 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:11.744145    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8\": container with ID starting with 38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8 not found: ID does not exist" containerID="38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8"
Oct 30 01:19:11.744230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.744188    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8"} err="failed to get container status \"38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8\": rpc error: code = NotFound desc = could not find container \"38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8\": container with ID starting with 38606d4bb243d28b93f03e9add2bcebaa82d58eb278e2764aac5fa2ee4f1e1e8 not found: ID does not exist"
Oct 30 01:19:11.776424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.776387    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"]
Oct 30 01:19:11.781129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.781096    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0e5711/test-model-0e5711-kserve-567f756789-9vkbg"]
Oct 30 01:19:11.934167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:11.934138    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:19:12.006597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.006508    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab79d576-7543-4573-bfca-2cfa6a6000d6-kube-api-access-9zlch\" (UniqueName: \"kubernetes.io/projected/ab79d576-7543-4573-bfca-2cfa6a6000d6-kube-api-access-9zlch\") pod \"ab79d576-7543-4573-bfca-2cfa6a6000d6\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") "
Oct 30 01:19:12.006791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.006767    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab79d576-7543-4573-bfca-2cfa6a6000d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab79d576-7543-4573-bfca-2cfa6a6000d6-plugins-config-volume\") pod \"ab79d576-7543-4573-bfca-2cfa6a6000d6\" (UID: \"ab79d576-7543-4573-bfca-2cfa6a6000d6\") "
Oct 30 01:19:12.007211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.007183    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab79d576-7543-4573-bfca-2cfa6a6000d6-plugins-config-volume" pod "ab79d576-7543-4573-bfca-2cfa6a6000d6" (UID: "ab79d576-7543-4573-bfca-2cfa6a6000d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:12.007359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.007343    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab79d576-7543-4573-bfca-2cfa6a6000d6-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:12.008875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.008841    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab79d576-7543-4573-bfca-2cfa6a6000d6-kube-api-access-9zlch" pod "ab79d576-7543-4573-bfca-2cfa6a6000d6" (UID: "ab79d576-7543-4573-bfca-2cfa6a6000d6"). InnerVolumeSpecName "kube-api-access-9zlch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:12.109671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.109626    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zlch\" (UniqueName: \"kubernetes.io/projected/ab79d576-7543-4573-bfca-2cfa6a6000d6-kube-api-access-9zlch\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:12.583912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.583873    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d9557d-48f8-409a-9cbc-22b8a39739c3" path="/var/lib/kubelet/pods/86d9557d-48f8-409a-9cbc-22b8a39739c3/volumes"
Oct 30 01:19:12.584410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.584395    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" path="/var/lib/kubelet/pods/e0d7b9ba-12aa-4de5-b9d2-edba23bcf767/volumes"
Oct 30 01:19:12.735300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.735260    5104 generic.go:356] "Generic (PLEG): container finished" podID="ab79d576-7543-4573-bfca-2cfa6a6000d6" containerID="c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1" exitCode=0
Oct 30 01:19:12.735537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.735437    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz" event={"ID":"ab79d576-7543-4573-bfca-2cfa6a6000d6","Type":"ContainerDied","Data":"c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1"}
Oct 30 01:19:12.735618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.735544    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz" event={"ID":"ab79d576-7543-4573-bfca-2cfa6a6000d6","Type":"ContainerDied","Data":"daf804abfc88bc2e73e030497110943855d8f11c96a50779dd1850c963f83bb4"}
Oct 30 01:19:12.735618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.735577    5104 scope.go:122] "RemoveContainer" containerID="c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1"
Oct 30 01:19:12.735805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.735718    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"
Oct 30 01:19:12.739629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.739595    5104 generic.go:356] "Generic (PLEG): container finished" podID="9f40dd84-6d57-45f8-80c7-ada42b04e41c" containerID="972d54775caaae2de591865a0af937cd9d67fe8d56a1b69fb2831d8e3a7e9cde" exitCode=0
Oct 30 01:19:12.739820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.739715    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs" event={"ID":"9f40dd84-6d57-45f8-80c7-ada42b04e41c","Type":"ContainerDied","Data":"972d54775caaae2de591865a0af937cd9d67fe8d56a1b69fb2831d8e3a7e9cde"}
Oct 30 01:19:12.777343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.768663    5104 scope.go:122] "RemoveContainer" containerID="c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1"
Oct 30 01:19:12.777343 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:12.770067    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1\": container with ID starting with c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1 not found: ID does not exist" containerID="c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1"
Oct 30 01:19:12.777343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.770105    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1"} err="failed to get container status \"c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1\": rpc error: code = NotFound desc = could not find container \"c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1\": container with ID starting with c53e88e9422798ea043e4383f5f0f07230951d93938f5d61b7f241889c2bf9d1 not found: ID does not exist"
Oct 30 01:19:12.777343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.776705    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"]
Oct 30 01:19:12.777343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.776944    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy" containerID="cri-o://80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29" gracePeriod=30
Oct 30 01:19:12.830600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.830561    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"]
Oct 30 01:19:12.835235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:12.835158    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50d078-58ffdcc78d-qh9nz"]
Oct 30 01:19:13.397537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:13.394698    5104 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:19:13.397537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:13.394762    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:19:13.962016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:13.961980    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:14.032491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.032427    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:19:14.065028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.064986    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f40dd84-6d57-45f8-80c7-ada42b04e41c-kube-api-access-zkrcg\" (UniqueName: \"kubernetes.io/projected/9f40dd84-6d57-45f8-80c7-ada42b04e41c-kube-api-access-zkrcg\") pod \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") "
Oct 30 01:19:14.065239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.065186    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-openshift-ca\") pod \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") "
Oct 30 01:19:14.065390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.065370    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-xks-ca\") pod \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\" (UID: \"9f40dd84-6d57-45f8-80c7-ada42b04e41c\") "
Oct 30 01:19:14.065871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.065844    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-openshift-ca" pod "9f40dd84-6d57-45f8-80c7-ada42b04e41c" (UID: "9f40dd84-6d57-45f8-80c7-ada42b04e41c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:14.068174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.068143    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f40dd84-6d57-45f8-80c7-ada42b04e41c-kube-api-access-zkrcg" pod "9f40dd84-6d57-45f8-80c7-ada42b04e41c" (UID: "9f40dd84-6d57-45f8-80c7-ada42b04e41c"). InnerVolumeSpecName "kube-api-access-zkrcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:14.068495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.068428    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-xks-ca" pod "9f40dd84-6d57-45f8-80c7-ada42b04e41c" (UID: "9f40dd84-6d57-45f8-80c7-ada42b04e41c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:14.166966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.166926    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-envoy\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.166993    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istiod-ca-cert\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167016    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-token\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-token\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167090    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-podinfo\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167131    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-data\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167192    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-certs\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167274    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-socket\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167306    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-kube-api-access-fr2qf\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-kube-api-access-fr2qf\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167436    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istiod-ca-cert" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:14.167603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167445    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-credential-socket\") pod \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\" (UID: \"952fc7a6-e26d-4361-ae4a-d2ce13284a30\") "
Oct 30 01:19:14.167729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167613    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-socket" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:14.167729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167648    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-certs" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:14.167729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167686    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-credential-socket" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:14.167895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.167809    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-data" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:14.168359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168338    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.168359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168360    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrcg\" (UniqueName: \"kubernetes.io/projected/9f40dd84-6d57-45f8-80c7-ada42b04e41c-kube-api-access-zkrcg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.168592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168370    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.168592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168379    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.168592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168388    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.168592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168396    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.168592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168406    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.168592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.168414    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f40dd84-6d57-45f8-80c7-ada42b04e41c-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.169412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.169389    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-podinfo" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:14.169530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.169482    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-envoy" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:14.169870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.169848    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-token" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:14.169929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.169904    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-kube-api-access-fr2qf" pod "952fc7a6-e26d-4361-ae4a-d2ce13284a30" (UID: "952fc7a6-e26d-4361-ae4a-d2ce13284a30"). InnerVolumeSpecName "kube-api-access-fr2qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:14.270059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.269966    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.270059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.270004    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.270059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.270014    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952fc7a6-e26d-4361-ae4a-d2ce13284a30-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.270059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.270023    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2qf\" (UniqueName: \"kubernetes.io/projected/952fc7a6-e26d-4361-ae4a-d2ce13284a30-kube-api-access-fr2qf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:14.584551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.584442    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab79d576-7543-4573-bfca-2cfa6a6000d6" path="/var/lib/kubelet/pods/ab79d576-7543-4573-bfca-2cfa6a6000d6/volumes"
Oct 30 01:19:14.760318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.760284    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"
Oct 30 01:19:14.760572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.760314    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs" event={"ID":"9f40dd84-6d57-45f8-80c7-ada42b04e41c","Type":"ContainerDied","Data":"cc6cc17bb78c7f0f6bf11345ad0e4702ac00f2448248d34467b4d3307637604e"}
Oct 30 01:19:14.760572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.760364    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6cc17bb78c7f0f6bf11345ad0e4702ac00f2448248d34467b4d3307637604e"
Oct 30 01:19:14.762021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.761996    5104 generic.go:356] "Generic (PLEG): container finished" podID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerID="2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438" exitCode=0
Oct 30 01:19:14.762180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.762041    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"
Oct 30 01:19:14.762180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.762082    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" event={"ID":"952fc7a6-e26d-4361-ae4a-d2ce13284a30","Type":"ContainerDied","Data":"2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438"}
Oct 30 01:19:14.762180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.762134    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk" event={"ID":"952fc7a6-e26d-4361-ae4a-d2ce13284a30","Type":"ContainerDied","Data":"56a0e46dbd678ca7698e0e2838a5291b20c9e1b1761c9b19241fbcefbbd8eaa0"}
Oct 30 01:19:14.762180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.762160    5104 scope.go:122] "RemoveContainer" containerID="2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438"
Oct 30 01:19:14.777574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.777553    5104 scope.go:122] "RemoveContainer" containerID="2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438"
Oct 30 01:19:14.777976 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:14.777944    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438\": container with ID starting with 2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438 not found: ID does not exist" containerID="2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438"
Oct 30 01:19:14.778129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.777987    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438"} err="failed to get container status \"2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438\": rpc error: code = NotFound desc = could not find container \"2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438\": container with ID starting with 2dced900c02fd6ce909ddcafb7eb8f7bb53c0115ff3bdabef8f517d700a3e438 not found: ID does not exist"
Oct 30 01:19:14.828296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.828262    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"]
Oct 30 01:19:14.839680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:14.839597    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-4q4sk"]
Oct 30 01:19:15.032495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:15.026884    5104 patch_prober.go:29] interesting pod/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:19:15.032495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:15.026946    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:19:15.232036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:15.232000    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" podUID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:16.224669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.224630    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"]
Oct 30 01:19:16.225195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.224893    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" containerID="cri-o://4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c" gracePeriod=30
Oct 30 01:19:16.643929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.643823    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" path="/var/lib/kubelet/pods/952fc7a6-e26d-4361-ae4a-d2ce13284a30/volumes"
Oct 30 01:19:16.698759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.698725    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"]
Oct 30 01:19:16.699565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699547    5104 cpu_manager.go:414] "containerMap: removing container" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy"
Oct 30 01:19:16.699668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699566    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy"
Oct 30 01:19:16.699668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699585    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ab79d576-7543-4573-bfca-2cfa6a6000d6" containerName="payload-processing"
Oct 30 01:19:16.699668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699590    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab79d576-7543-4573-bfca-2cfa6a6000d6" containerName="payload-processing"
Oct 30 01:19:16.699668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699617    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9f40dd84-6d57-45f8-80c7-ada42b04e41c" containerName="revoke-keys"
Oct 30 01:19:16.699668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699624    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f40dd84-6d57-45f8-80c7-ada42b04e41c" containerName="revoke-keys"
Oct 30 01:19:16.699668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699638    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" containerName="main"
Oct 30 01:19:16.699668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699643    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" containerName="main"
Oct 30 01:19:16.699903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699757    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="952fc7a6-e26d-4361-ae4a-d2ce13284a30" containerName="istio-proxy"
Oct 30 01:19:16.699903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699767    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab79d576-7543-4573-bfca-2cfa6a6000d6" containerName="payload-processing"
Oct 30 01:19:16.699903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699777    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f40dd84-6d57-45f8-80c7-ada42b04e41c" containerName="revoke-keys"
Oct 30 01:19:16.699903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.699785    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d7b9ba-12aa-4de5-b9d2-edba23bcf767" containerName="main"
Oct 30 01:19:16.705124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.705094    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.720089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.719304    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:19:16.720291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.720266    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:19:16.723448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.723421    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"]
Oct 30 01:19:16.809055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.809008    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:16.809239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.809065    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:19:16.832903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.832856    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.833086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.832909    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8v9d\" (UniqueName: \"kubernetes.io/projected/fd0aca39-ce41-450d-9d41-74a7a4eabd10-kube-api-access-k8v9d\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.833086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.833065    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.934687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.934650    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.934687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.934690    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8v9d\" (UniqueName: \"kubernetes.io/projected/fd0aca39-ce41-450d-9d41-74a7a4eabd10-kube-api-access-k8v9d\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.935004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.934760    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.935441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.935408    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.937235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.937213    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:16.953880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:16.953841    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8v9d\" (UniqueName: \"kubernetes.io/projected/fd0aca39-ce41-450d-9d41-74a7a4eabd10-kube-api-access-k8v9d\") pod \"maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:17.023018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:17.022980    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:17.342737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:17.342703    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"]
Oct 30 01:19:17.343176 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:17.343025    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0aca39_ce41_450d_9d41_74a7a4eabd10.slice/crio-4221607545b7c5d43b7d8b1eb0cb055a1be7156ca9c474e5334155dfeddec179 WatchSource:0}: Error finding container 4221607545b7c5d43b7d8b1eb0cb055a1be7156ca9c474e5334155dfeddec179: Status 404 returned error can't find the container with id 4221607545b7c5d43b7d8b1eb0cb055a1be7156ca9c474e5334155dfeddec179
Oct 30 01:19:17.801527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:17.801414    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7" event={"ID":"fd0aca39-ce41-450d-9d41-74a7a4eabd10","Type":"ContainerStarted","Data":"4221607545b7c5d43b7d8b1eb0cb055a1be7156ca9c474e5334155dfeddec179"}
Oct 30 01:19:17.842736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:17.842676    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7" podStartSLOduration=1.842660642 podStartE2EDuration="1.842660642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:17.840398852 +0000 UTC m=+2199.388089349" watchObservedRunningTime="2026-10-30 01:19:17.842660642 +0000 UTC m=+2199.390351138"
Oct 30 01:19:18.105505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.105440    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:19:18.259621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259580    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-credential-socket\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.259835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259658    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istiod-ca-cert\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.259835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259709    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-kube-api-access-9wpp2\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-kube-api-access-9wpp2\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.259835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259764    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-podinfo\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.259835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259793    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-certs\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.260068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259898    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-data\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.260068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259931    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-token\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-token\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.260068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.259993    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-envoy\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.260219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.260159    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-socket\") pod \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\" (UID: \"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a\") "
Oct 30 01:19:18.261030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.260988    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-socket" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:18.261237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.261219    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-credential-socket" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:18.261701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.261679    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istiod-ca-cert" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:18.261799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.261755    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-certs" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:18.265512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.262670    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-data" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:18.272307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.272245    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-podinfo" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:18.272739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.272690    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-envoy" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:18.273105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.273017    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-token" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:18.326853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.326805    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-kube-api-access-9wpp2" pod "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" (UID: "3e5d5e2e-e9b2-4ff2-957a-f736ea96503a"). InnerVolumeSpecName "kube-api-access-9wpp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:18.362651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362619    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.362651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362647    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.362651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362656    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.363128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362665    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.363128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362673    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.363128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362680    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.363128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362688    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.363128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362697    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.363128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.362706    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wpp2\" (UniqueName: \"kubernetes.io/projected/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a-kube-api-access-9wpp2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:18.478549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.476754    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"]
Oct 30 01:19:18.478549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.477435    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy" containerID="cri-o://2e4ff9b021a141cc9ce2db8b50ea1d14f13ac0770b35b481731631dad538e689" gracePeriod=30
Oct 30 01:19:18.849958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.849908    5104 generic.go:356] "Generic (PLEG): container finished" podID="fd0aca39-ce41-450d-9d41-74a7a4eabd10" containerID="a54479e95cb2b55785b75a1cfdb017820ea73fe0095532c66d326a148719481c" exitCode=0
Oct 30 01:19:18.852931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.852890    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7" event={"ID":"fd0aca39-ce41-450d-9d41-74a7a4eabd10","Type":"ContainerDied","Data":"a54479e95cb2b55785b75a1cfdb017820ea73fe0095532c66d326a148719481c"}
Oct 30 01:19:18.855410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.854415    5104 generic.go:356] "Generic (PLEG): container finished" podID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerID="80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29" exitCode=0
Oct 30 01:19:18.855410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.854512    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"
Oct 30 01:19:18.855410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.854522    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" event={"ID":"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a","Type":"ContainerDied","Data":"80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29"}
Oct 30 01:19:18.855410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.854565    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26" event={"ID":"3e5d5e2e-e9b2-4ff2-957a-f736ea96503a","Type":"ContainerDied","Data":"61ec9ce98a7bd57a0745dca8fb2000a17da6e8e59cea5fe799ba1320c3e30d4e"}
Oct 30 01:19:18.855410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.854593    5104 scope.go:122] "RemoveContainer" containerID="80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29"
Oct 30 01:19:18.875550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.874563    5104 scope.go:122] "RemoveContainer" containerID="f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767"
Oct 30 01:19:18.891758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.891714    5104 scope.go:122] "RemoveContainer" containerID="80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29"
Oct 30 01:19:18.892239 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:18.892209    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29\": container with ID starting with 80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29 not found: ID does not exist" containerID="80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29"
Oct 30 01:19:18.892355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.892253    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29"} err="failed to get container status \"80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29\": rpc error: code = NotFound desc = could not find container \"80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29\": container with ID starting with 80e7487388b11723ee07dedc3460866b75930dd3de4f32c2696c86c052c9ed29 not found: ID does not exist"
Oct 30 01:19:18.892355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.892273    5104 scope.go:122] "RemoveContainer" containerID="f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767"
Oct 30 01:19:18.892688 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:18.892616    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767\": container with ID starting with f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767 not found: ID does not exist" containerID="f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767"
Oct 30 01:19:18.892772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.892700    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767"} err="failed to get container status \"f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767\": rpc error: code = NotFound desc = could not find container \"f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767\": container with ID starting with f2157802e8c1e8f26023996ba1629328710aca7e3dcb49e8210f72977a766767 not found: ID does not exist"
Oct 30 01:19:18.926056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.925985    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"]
Oct 30 01:19:18.929446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.929407    5104 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:19:18.929701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.929529    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:19:18.931548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:18.931520    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-56d1d9-openshift-default-5f88488bc9-85f26"]
Oct 30 01:19:19.927518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:19.926527    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"]
Oct 30 01:19:19.996601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:19.995340    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"]
Oct 30 01:19:20.145193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.145154    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:20.260722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.260370    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-xks-ca\") pod \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") "
Oct 30 01:19:20.260722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.260646    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0aca39-ce41-450d-9d41-74a7a4eabd10-kube-api-access-k8v9d\" (UniqueName: \"kubernetes.io/projected/fd0aca39-ce41-450d-9d41-74a7a4eabd10-kube-api-access-k8v9d\") pod \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") "
Oct 30 01:19:20.261010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.260906    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-openshift-ca\") pod \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\" (UID: \"fd0aca39-ce41-450d-9d41-74a7a4eabd10\") "
Oct 30 01:19:20.298598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.298521    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-openshift-ca" pod "fd0aca39-ce41-450d-9d41-74a7a4eabd10" (UID: "fd0aca39-ce41-450d-9d41-74a7a4eabd10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:20.298793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.298623    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-xks-ca" pod "fd0aca39-ce41-450d-9d41-74a7a4eabd10" (UID: "fd0aca39-ce41-450d-9d41-74a7a4eabd10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:20.305098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.304300    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:20.305098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.304338    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0aca39-ce41-450d-9d41-74a7a4eabd10-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:20.317125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.317083    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0aca39-ce41-450d-9d41-74a7a4eabd10-kube-api-access-k8v9d" pod "fd0aca39-ce41-450d-9d41-74a7a4eabd10" (UID: "fd0aca39-ce41-450d-9d41-74a7a4eabd10"). InnerVolumeSpecName "kube-api-access-k8v9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:20.405576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.405476    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8v9d\" (UniqueName: \"kubernetes.io/projected/fd0aca39-ce41-450d-9d41-74a7a4eabd10-kube-api-access-k8v9d\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:20.582412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.582368    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" path="/var/lib/kubelet/pods/3e5d5e2e-e9b2-4ff2-957a-f736ea96503a/volumes"
Oct 30 01:19:20.891896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.891864    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"
Oct 30 01:19:20.892735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.892244    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7" event={"ID":"fd0aca39-ce41-450d-9d41-74a7a4eabd10","Type":"ContainerDied","Data":"4221607545b7c5d43b7d8b1eb0cb055a1be7156ca9c474e5334155dfeddec179"}
Oct 30 01:19:20.893031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:20.893005    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4221607545b7c5d43b7d8b1eb0cb055a1be7156ca9c474e5334155dfeddec179"
Oct 30 01:19:21.148062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.147966    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"]
Oct 30 01:19:21.149172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.149148    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fd0aca39-ce41-450d-9d41-74a7a4eabd10" containerName="revoke-keys"
Oct 30 01:19:21.149172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.149173    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0aca39-ce41-450d-9d41-74a7a4eabd10" containerName="revoke-keys"
Oct 30 01:19:21.149313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.149187    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy"
Oct 30 01:19:21.149313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.149193    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy"
Oct 30 01:19:21.149428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.149332    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy"
Oct 30 01:19:21.149428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.149347    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0aca39-ce41-450d-9d41-74a7a4eabd10" containerName="revoke-keys"
Oct 30 01:19:21.149428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.149354    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy"
Oct 30 01:19:21.154324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.154246    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.168505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.168473    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-262230\"/\"openshift-service-ca.crt\""
Oct 30 01:19:21.168662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.168551    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-262230\"/\"kube-root-ca.crt\""
Oct 30 01:19:21.175292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.175257    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"]
Oct 30 01:19:21.180515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.179294    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-262230\"/\"rate-test-model-262230-kserve-self-signed-certs\""
Oct 30 01:19:21.180515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.179315    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-262230\"/\"default-dockercfg-snv7d\""
Oct 30 01:19:21.223910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.223873    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-dshm\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.224244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.224221    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-tmp-dir\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.224603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.224579    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-home\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.224714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.224695    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a3c713-4ccd-4243-98ac-59b9e73b2823-tls-certs\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.224885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.224866    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6h4\" (UniqueName: \"kubernetes.io/projected/88a3c713-4ccd-4243-98ac-59b9e73b2823-kube-api-access-cd6h4\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.224963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.224902    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-model-cache\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326210    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-home\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326299    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a3c713-4ccd-4243-98ac-59b9e73b2823-tls-certs\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326378    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6h4\" (UniqueName: \"kubernetes.io/projected/88a3c713-4ccd-4243-98ac-59b9e73b2823-kube-api-access-cd6h4\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326397    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-model-cache\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326438    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-dshm\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326545    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-tmp-dir\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326685    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-home\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326806    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-model-cache\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.326940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.326923    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-tmp-dir\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.328846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.328818    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-dshm\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.329129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.329110    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a3c713-4ccd-4243-98ac-59b9e73b2823-tls-certs\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.344912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.344884    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6h4\" (UniqueName: \"kubernetes.io/projected/88a3c713-4ccd-4243-98ac-59b9e73b2823-kube-api-access-cd6h4\") pod \"rate-test-model-262230-kserve-955d7d757-f4ld2\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") " pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.472491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.472302    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:21.783601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.783565    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:19:21.903026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.902984    5104 generic.go:356] "Generic (PLEG): container finished" podID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerID="4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c" exitCode=0
Oct 30 01:19:21.903245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.903044    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"
Oct 30 01:19:21.903245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.903075    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" event={"ID":"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f","Type":"ContainerDied","Data":"4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c"}
Oct 30 01:19:21.903245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.903143    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9" event={"ID":"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f","Type":"ContainerDied","Data":"71e4c609169fc113958a04bbce66997c2f7637bb0ffc42ea85507411182a7a19"}
Oct 30 01:19:21.903245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.903184    5104 scope.go:122] "RemoveContainer" containerID="4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c"
Oct 30 01:19:21.915212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.915186    5104 scope.go:122] "RemoveContainer" containerID="4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c"
Oct 30 01:19:21.915621 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:21.915595    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c\": container with ID starting with 4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c not found: ID does not exist" containerID="4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c"
Oct 30 01:19:21.915758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.915627    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c"} err="failed to get container status \"4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c\": rpc error: code = NotFound desc = could not find container \"4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c\": container with ID starting with 4011b2e854410330381177cb1ae8b72a7dd8349d762e30fef53552c07b3db20c not found: ID does not exist"
Oct 30 01:19:21.947745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.947673    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-podinfo\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.947929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.947749    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-certs\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.947929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.947792    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-data\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.947929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.947821    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-token\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-token\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.948124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.947974    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-credential-socket\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.948124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948026    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istiod-ca-cert\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.948124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948077    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-certs" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:21.948124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948088    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-socket\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.948124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948096    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-data" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:21.948124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948121    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-kube-api-access-fsnp8\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-kube-api-access-fsnp8\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.948449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948169    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-envoy\") pod \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\" (UID: \"b7086e18-9dcd-48b3-9f26-24aa1b24cb1f\") "
Oct 30 01:19:21.948449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948320    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-socket" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:21.948449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948417    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istiod-ca-cert" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:21.948642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948585    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-credential-socket" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:21.948928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948904    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:21.948928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948931    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:21.949095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948940    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:21.949095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948948    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:21.949095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.948957    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:21.950685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.950656    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-podinfo" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:21.951014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.950978    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-token" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:21.951430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.951370    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-kube-api-access-fsnp8" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "kube-api-access-fsnp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:21.951570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.951418    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-envoy" pod "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" (UID: "b7086e18-9dcd-48b3-9f26-24aa1b24cb1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:21.989316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:21.989277    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"]
Oct 30 01:19:21.990671 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:21.990597    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88a3c713_4ccd_4243_98ac_59b9e73b2823.slice/crio-53cfd23e6b5620a21aa8021e0cd8174f178c434981840b246de31785ac503ddc WatchSource:0}: Error finding container 53cfd23e6b5620a21aa8021e0cd8174f178c434981840b246de31785ac503ddc: Status 404 returned error can't find the container with id 53cfd23e6b5620a21aa8021e0cd8174f178c434981840b246de31785ac503ddc
Oct 30 01:19:22.050469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.050429    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsnp8\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-kube-api-access-fsnp8\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:22.050608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.050494    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:22.050608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.050505    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:22.050608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.050514    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:22.250565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.250427    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"]
Oct 30 01:19:22.255624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.255594    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-bn2p9"]
Oct 30 01:19:22.582540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.582431    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" path="/var/lib/kubelet/pods/b7086e18-9dcd-48b3-9f26-24aa1b24cb1f/volumes"
Oct 30 01:19:22.917769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.917666    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" event={"ID":"88a3c713-4ccd-4243-98ac-59b9e73b2823","Type":"ContainerStarted","Data":"df8040a68d12757b9fd8909a920ad3ad75ede73fcaa20989280553fd4116e726"}
Oct 30 01:19:22.917769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.917727    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" event={"ID":"88a3c713-4ccd-4243-98ac-59b9e73b2823","Type":"ContainerStarted","Data":"53cfd23e6b5620a21aa8021e0cd8174f178c434981840b246de31785ac503ddc"}
Oct 30 01:19:22.958759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:22.958695    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podStartSLOduration=1.9586762979999999 podStartE2EDuration="1.958676298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:22.954315191 +0000 UTC m=+2204.502005685" watchObservedRunningTime="2026-10-30 01:19:22.958676298 +0000 UTC m=+2204.506366796"
Oct 30 01:19:23.473435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:23.473402    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:23.931383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:23.931341    5104 generic.go:356] "Generic (PLEG): container finished" podID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerID="2e4ff9b021a141cc9ce2db8b50ea1d14f13ac0770b35b481731631dad538e689" exitCode=0
Oct 30 01:19:23.931627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:23.931413    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" event={"ID":"2b4d865c-5038-4871-8e4f-12cac274ca02","Type":"ContainerDied","Data":"2e4ff9b021a141cc9ce2db8b50ea1d14f13ac0770b35b481731631dad538e689"}
Oct 30 01:19:23.931627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:23.931503    5104 scope.go:122] "RemoveContainer" containerID="fedc832c0aa6907b55353184d1a82bfb260f71c7c32e406aa568c347167d10e7"
Oct 30 01:19:24.180432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.180392    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:19:24.289684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.289574    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-data\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.289684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.289643    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-kube-api-access-rcsxh\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-kube-api-access-rcsxh\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.289935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.289728    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-podinfo\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.289935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.289766    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-token\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-token\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.289935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.289869    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-certs\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.289935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.289916    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-socket\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.290174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.289983    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-credential-socket\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.290174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.290017    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4d865c-5038-4871-8e4f-12cac274ca02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4d865c-5038-4871-8e4f-12cac274ca02-istiod-ca-cert\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.290174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.290055    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-envoy\") pod \"2b4d865c-5038-4871-8e4f-12cac274ca02\" (UID: \"2b4d865c-5038-4871-8e4f-12cac274ca02\") "
Oct 30 01:19:24.291196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.290825    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-socket" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:24.291196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.290973    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.291413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.291360    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-credential-socket" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:24.291607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.291582    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-certs" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:24.291981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.291949    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4d865c-5038-4871-8e4f-12cac274ca02-istiod-ca-cert" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:24.292363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.292322    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-data" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:24.295311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.295286    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-podinfo" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:24.296869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.296847    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-token" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:24.297064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.297029    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-kube-api-access-rcsxh" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "kube-api-access-rcsxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:24.298523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.298434    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-envoy" pod "2b4d865c-5038-4871-8e4f-12cac274ca02" (UID: "2b4d865c-5038-4871-8e4f-12cac274ca02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:24.392554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392513    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.392554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392550    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcsxh\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-kube-api-access-rcsxh\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.392554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392565    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.392821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392577    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.392821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392590    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.392821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392602    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.392821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392615    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4d865c-5038-4871-8e4f-12cac274ca02-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.392821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.392627    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4d865c-5038-4871-8e4f-12cac274ca02-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:24.931477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.928261    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n" podUID="afe71f7b-8700-4284-b72d-1af3a263cd4c" containerName="payload-processing" containerID="cri-o://f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657" gracePeriod=25
Oct 30 01:19:24.998773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:24.989626    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:19:25.007485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.004584    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" podUID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" containerName="payload-pre-processing" containerID="cri-o://2aa8b19e18604d8e040a7b935d179eb349d551e210c9060eeda28e7e6b15c4d7" gracePeriod=25
Oct 30 01:19:25.036486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.027155    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"
Oct 30 01:19:25.062479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.029365    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54" event={"ID":"2b4d865c-5038-4871-8e4f-12cac274ca02","Type":"ContainerDied","Data":"3a522a955fb2ee5a8d99abdc31375a80fa87f33aa4d4cd1db094c9475ff6c2c3"}
Oct 30 01:19:25.062479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.050481    5104 scope.go:122] "RemoveContainer" containerID="2e4ff9b021a141cc9ce2db8b50ea1d14f13ac0770b35b481731631dad538e689"
Oct 30 01:19:25.173502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.173266    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"]
Oct 30 01:19:25.184024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.183916    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-34e896-openshift-default-6c7475f655-grr54"]
Oct 30 01:19:25.232485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.232277    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" podUID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:25.336197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.336169    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:19:25.452354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.452265    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe71f7b-8700-4284-b72d-1af3a263cd4c-kube-api-access-krhg6\" (UniqueName: \"kubernetes.io/projected/afe71f7b-8700-4284-b72d-1af3a263cd4c-kube-api-access-krhg6\") pod \"afe71f7b-8700-4284-b72d-1af3a263cd4c\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") "
Oct 30 01:19:25.452529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.452446    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe71f7b-8700-4284-b72d-1af3a263cd4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe71f7b-8700-4284-b72d-1af3a263cd4c-plugins-config-volume\") pod \"afe71f7b-8700-4284-b72d-1af3a263cd4c\" (UID: \"afe71f7b-8700-4284-b72d-1af3a263cd4c\") "
Oct 30 01:19:25.452870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.452838    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe71f7b-8700-4284-b72d-1af3a263cd4c-plugins-config-volume" pod "afe71f7b-8700-4284-b72d-1af3a263cd4c" (UID: "afe71f7b-8700-4284-b72d-1af3a263cd4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:25.453014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.453001    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe71f7b-8700-4284-b72d-1af3a263cd4c-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:25.455019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.454991    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe71f7b-8700-4284-b72d-1af3a263cd4c-kube-api-access-krhg6" pod "afe71f7b-8700-4284-b72d-1af3a263cd4c" (UID: "afe71f7b-8700-4284-b72d-1af3a263cd4c"). InnerVolumeSpecName "kube-api-access-krhg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:25.553932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:25.553895    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krhg6\" (UniqueName: \"kubernetes.io/projected/afe71f7b-8700-4284-b72d-1af3a263cd4c-kube-api-access-krhg6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:26.033322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.033281    5104 generic.go:356] "Generic (PLEG): container finished" podID="afe71f7b-8700-4284-b72d-1af3a263cd4c" containerID="f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657" exitCode=0
Oct 30 01:19:26.033870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.033328    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"
Oct 30 01:19:26.033870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.033362    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n" event={"ID":"afe71f7b-8700-4284-b72d-1af3a263cd4c","Type":"ContainerDied","Data":"f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657"}
Oct 30 01:19:26.033870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.033418    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n" event={"ID":"afe71f7b-8700-4284-b72d-1af3a263cd4c","Type":"ContainerDied","Data":"2e53b969ec4b334b6ab754bc50ca6f71be1ae04447bca419976686ce859fa465"}
Oct 30 01:19:26.033870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.033449    5104 scope.go:122] "RemoveContainer" containerID="f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657"
Oct 30 01:19:26.047425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.047400    5104 scope.go:122] "RemoveContainer" containerID="f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657"
Oct 30 01:19:26.047787 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:26.047761    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657\": container with ID starting with f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657 not found: ID does not exist" containerID="f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657"
Oct 30 01:19:26.047891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.047794    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657"} err="failed to get container status \"f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657\": rpc error: code = NotFound desc = could not find container \"f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657\": container with ID starting with f5cf962b94fae683c8e5a5c8cf8038482c6455fffade7ddd939ee88a0e932657 not found: ID does not exist"
Oct 30 01:19:26.083604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.083570    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"]
Oct 30 01:19:26.088071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.088040    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e69dee-f496f5d46-npq8n"]
Oct 30 01:19:26.581591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.581556    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" path="/var/lib/kubelet/pods/2b4d865c-5038-4871-8e4f-12cac274ca02/volumes"
Oct 30 01:19:26.582188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:26.582170    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe71f7b-8700-4284-b72d-1af3a263cd4c" path="/var/lib/kubelet/pods/afe71f7b-8700-4284-b72d-1af3a263cd4c/volumes"
Oct 30 01:19:27.205785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:27.205749    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"]
Oct 30 01:19:27.241281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:27.241243    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"]
Oct 30 01:19:29.147591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:29.147553    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" podUID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:31.473209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.473179    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:31.487436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.487403    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:31.882602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.882572    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:19:31.888078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.888039    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079b3bb9-71bb-4c5a-9bf9-7d509162e665-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079b3bb9-71bb-4c5a-9bf9-7d509162e665-plugins-config-volume\") pod \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") "
Oct 30 01:19:31.888283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.888150    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079b3bb9-71bb-4c5a-9bf9-7d509162e665-kube-api-access-78xxs\" (UniqueName: \"kubernetes.io/projected/079b3bb9-71bb-4c5a-9bf9-7d509162e665-kube-api-access-78xxs\") pod \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\" (UID: \"079b3bb9-71bb-4c5a-9bf9-7d509162e665\") "
Oct 30 01:19:31.888447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.888411    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079b3bb9-71bb-4c5a-9bf9-7d509162e665-plugins-config-volume" pod "079b3bb9-71bb-4c5a-9bf9-7d509162e665" (UID: "079b3bb9-71bb-4c5a-9bf9-7d509162e665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:31.889292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.889269    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079b3bb9-71bb-4c5a-9bf9-7d509162e665-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:31.890849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:31.890762    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079b3bb9-71bb-4c5a-9bf9-7d509162e665-kube-api-access-78xxs" pod "079b3bb9-71bb-4c5a-9bf9-7d509162e665" (UID: "079b3bb9-71bb-4c5a-9bf9-7d509162e665"). InnerVolumeSpecName "kube-api-access-78xxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:32.024093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.023939    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xxs\" (UniqueName: \"kubernetes.io/projected/079b3bb9-71bb-4c5a-9bf9-7d509162e665-kube-api-access-78xxs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:32.131672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.131638    5104 generic.go:356] "Generic (PLEG): container finished" podID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" containerID="6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335" exitCode=0
Oct 30 01:19:32.131874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.131683    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"
Oct 30 01:19:32.131874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.131725    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" event={"ID":"079b3bb9-71bb-4c5a-9bf9-7d509162e665","Type":"ContainerDied","Data":"6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335"}
Oct 30 01:19:32.131874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.131779    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp" event={"ID":"079b3bb9-71bb-4c5a-9bf9-7d509162e665","Type":"ContainerDied","Data":"ecad7a2f352dbbdc10be1573403b11791fb719b40b495ca83af4651b8a9fa8b8"}
Oct 30 01:19:32.131874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.131802    5104 scope.go:122] "RemoveContainer" containerID="6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335"
Oct 30 01:19:32.143817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.143795    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:19:32.144835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.144819    5104 scope.go:122] "RemoveContainer" containerID="6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335"
Oct 30 01:19:32.145129 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:32.145108    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335\": container with ID starting with 6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335 not found: ID does not exist" containerID="6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335"
Oct 30 01:19:32.145190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.145141    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335"} err="failed to get container status \"6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335\": rpc error: code = NotFound desc = could not find container \"6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335\": container with ID starting with 6fb4d209b210669bf8ab0e3fda49d5eea2231be18fd0c9577c44fdf42358b335 not found: ID does not exist"
Oct 30 01:19:32.182494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.182445    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"]
Oct 30 01:19:32.193488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.193440    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50d078-7cb9f878f9-wc6qp"]
Oct 30 01:19:32.206559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.206513    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" containerName="payload-processing" containerID="cri-o://d64b4f7afd768ca9cb2f0e90196fe2ea81c2371e6b7c629c6a43b6ced6b1ade7" gracePeriod=25
Oct 30 01:19:32.241586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.241542    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerName="payload-pre-processing" containerID="cri-o://6b7f6afa7887eba8c3cabb4e85d1195ef4b73f54f8860d16fb48008116715d0b" gracePeriod=25
Oct 30 01:19:32.362531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.362409    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:32.362531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.362444    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:32.581257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:32.581221    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" path="/var/lib/kubelet/pods/079b3bb9-71bb-4c5a-9bf9-7d509162e665/volumes"
Oct 30 01:19:36.558357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.558313    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"]
Oct 30 01:19:36.559155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559133    5104 cpu_manager.go:414] "containerMap: removing container" podUID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" containerName="payload-pre-processing"
Oct 30 01:19:36.559155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559154    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" containerName="payload-pre-processing"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559170    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559176    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559209    5104 cpu_manager.go:414] "containerMap: removing container" podUID="afe71f7b-8700-4284-b72d-1af3a263cd4c" containerName="payload-processing"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559215    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe71f7b-8700-4284-b72d-1af3a263cd4c" containerName="payload-processing"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559237    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559242    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559253    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559258    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5d5e2e-e9b2-4ff2-957a-f736ea96503a" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559397    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe71f7b-8700-4284-b72d-1af3a263cd4c" containerName="payload-processing"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559480    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7086e18-9dcd-48b3-9f26-24aa1b24cb1f" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559502    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy"
Oct 30 01:19:36.562682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.559540    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="079b3bb9-71bb-4c5a-9bf9-7d509162e665" containerName="payload-pre-processing"
Oct 30 01:19:36.567290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.565580    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.587217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.585998    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-18a189\"/\"auth-test-model-18a189-kserve-self-signed-certs\""
Oct 30 01:19:36.587835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.587646    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-18a189\"/\"openshift-service-ca.crt\""
Oct 30 01:19:36.587835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.587761    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-18a189\"/\"kube-root-ca.crt\""
Oct 30 01:19:36.595724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.595692    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"]
Oct 30 01:19:36.596556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.596177    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-18a189\"/\"default-dockercfg-5q66p\""
Oct 30 01:19:36.721530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.721485    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ca880f-cacc-42b7-9250-429c367c4f15-tls-certs\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.721758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.721652    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-tmp-dir\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.721851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.721829    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4d5\" (UniqueName: \"kubernetes.io/projected/93ca880f-cacc-42b7-9250-429c367c4f15-kube-api-access-th4d5\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.721919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.721876    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-dshm\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.721986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.721961    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-home\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.722237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.722212    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-model-cache\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.823864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.823778    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-model-cache\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.823914    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ca880f-cacc-42b7-9250-429c367c4f15-tls-certs\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.823969    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-tmp-dir\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.824070    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4d5\" (UniqueName: \"kubernetes.io/projected/93ca880f-cacc-42b7-9250-429c367c4f15-kube-api-access-th4d5\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.824111    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-dshm\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.824138    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-home\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.824222    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-model-cache\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.824315    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-tmp-dir\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.824613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.824592    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-home\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.826618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.826593    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-dshm\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.826859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.826835    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ca880f-cacc-42b7-9250-429c367c4f15-tls-certs\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.845428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.845387    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4d5\" (UniqueName: \"kubernetes.io/projected/93ca880f-cacc-42b7-9250-429c367c4f15-kube-api-access-th4d5\") pod \"auth-test-model-18a189-kserve-7468b8f6df-dlqtp\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") " pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:36.935479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:36.916303    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:37.302452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:37.302414    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"]
Oct 30 01:19:37.310163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:37.310128    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e69dee-tjfzs"]
Oct 30 01:19:37.492766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:37.489780    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"]
Oct 30 01:19:37.499128 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:37.499089    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93ca880f_cacc_42b7_9250_429c367c4f15.slice/crio-35873ca080c744aa1bb911d102bc093fb296cadce86d1b22981e678e09fc2330 WatchSource:0}: Error finding container 35873ca080c744aa1bb911d102bc093fb296cadce86d1b22981e678e09fc2330: Status 404 returned error can't find the container with id 35873ca080c744aa1bb911d102bc093fb296cadce86d1b22981e678e09fc2330
Oct 30 01:19:38.196662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:38.196620    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" event={"ID":"93ca880f-cacc-42b7-9250-429c367c4f15","Type":"ContainerStarted","Data":"05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742"}
Oct 30 01:19:38.197188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:38.196674    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" event={"ID":"93ca880f-cacc-42b7-9250-429c367c4f15","Type":"ContainerStarted","Data":"35873ca080c744aa1bb911d102bc093fb296cadce86d1b22981e678e09fc2330"}
Oct 30 01:19:38.236491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:38.236421    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podStartSLOduration=2.236406097 podStartE2EDuration="2.236406097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:38.235139543 +0000 UTC m=+2219.782830040" watchObservedRunningTime="2026-10-30 01:19:38.236406097 +0000 UTC m=+2219.784096628"
Oct 30 01:19:38.585572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:38.585478    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f40dd84-6d57-45f8-80c7-ada42b04e41c" path="/var/lib/kubelet/pods/9f40dd84-6d57-45f8-80c7-ada42b04e41c/volumes"
Oct 30 01:19:38.924972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:38.924932    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:39.160544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:39.160058    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" podUID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:39.164348 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:39.164310    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:19:42.362404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:42.362350    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:42.362404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:42.362366    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:19:44.453698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:44.453641    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"]
Oct 30 01:19:44.454240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:44.453908    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" containerID="cri-o://0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d" gracePeriod=30
Oct 30 01:19:44.492324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:44.491795    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"]
Oct 30 01:19:44.498567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:44.498527    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e5711-fwpl7"]
Oct 30 01:19:44.582793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:44.582758    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0aca39-ce41-450d-9d41-74a7a4eabd10" path="/var/lib/kubelet/pods/fd0aca39-ce41-450d-9d41-74a7a4eabd10/volumes"
Oct 30 01:19:44.796709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:44.796293    5104 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:19:44.796709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:44.796365    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:19:45.324511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.315228    5104 generic.go:356] "Generic (PLEG): container finished" podID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" containerID="2aa8b19e18604d8e040a7b935d179eb349d551e210c9060eeda28e7e6b15c4d7" exitCode=0
Oct 30 01:19:45.324511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.315305    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" event={"ID":"7036635b-a1e2-4679-99dc-0e054dbbc2c5","Type":"ContainerDied","Data":"2aa8b19e18604d8e040a7b935d179eb349d551e210c9060eeda28e7e6b15c4d7"}
Oct 30 01:19:45.403024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.402994    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:19:45.545449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.545414    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7036635b-a1e2-4679-99dc-0e054dbbc2c5-kube-api-access-wfspj\" (UniqueName: \"kubernetes.io/projected/7036635b-a1e2-4679-99dc-0e054dbbc2c5-kube-api-access-wfspj\") pod \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") "
Oct 30 01:19:45.545903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.545557    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7036635b-a1e2-4679-99dc-0e054dbbc2c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7036635b-a1e2-4679-99dc-0e054dbbc2c5-plugins-config-volume\") pod \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\" (UID: \"7036635b-a1e2-4679-99dc-0e054dbbc2c5\") "
Oct 30 01:19:45.545987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.545958    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7036635b-a1e2-4679-99dc-0e054dbbc2c5-plugins-config-volume" pod "7036635b-a1e2-4679-99dc-0e054dbbc2c5" (UID: "7036635b-a1e2-4679-99dc-0e054dbbc2c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:45.546108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.546093    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7036635b-a1e2-4679-99dc-0e054dbbc2c5-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:45.548038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.548004    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7036635b-a1e2-4679-99dc-0e054dbbc2c5-kube-api-access-wfspj" pod "7036635b-a1e2-4679-99dc-0e054dbbc2c5" (UID: "7036635b-a1e2-4679-99dc-0e054dbbc2c5"). InnerVolumeSpecName "kube-api-access-wfspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:45.565400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.565347    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"]
Oct 30 01:19:45.566688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.566659    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" containerName="payload-pre-processing"
Oct 30 01:19:45.566688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.566688    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" containerName="payload-pre-processing"
Oct 30 01:19:45.566883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.566800    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy"
Oct 30 01:19:45.566883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.566810    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy"
Oct 30 01:19:45.567141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.567062    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" containerName="payload-pre-processing"
Oct 30 01:19:45.567141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.567094    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4d865c-5038-4871-8e4f-12cac274ca02" containerName="istio-proxy"
Oct 30 01:19:45.572469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.572426    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.593087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.590520    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e2d6d3-openshift-default-dockercfg-h487c\""
Oct 30 01:19:45.593981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.593953    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"]
Oct 30 01:19:45.647357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647253    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcm4\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-kube-api-access-gtcm4\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647339    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-credential-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647523    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-data\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647632    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647681    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-certs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647721    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-envoy\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647750    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647784    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-token\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.647910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.647894    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-podinfo\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.648098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.648085    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfspj\" (UniqueName: \"kubernetes.io/projected/7036635b-a1e2-4679-99dc-0e054dbbc2c5-kube-api-access-wfspj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761202    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcm4\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-kube-api-access-gtcm4\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761307    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-credential-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761395    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-data\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761475    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761524    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-certs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761552    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-envoy\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761580    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761612    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-token\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.761716    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-podinfo\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.766263    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.766922    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-credential-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.767162    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-data\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.768788    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-certs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.772482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.769040    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.805488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.804893    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-token\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.810492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.808289    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-envoy\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.836482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.834866    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-podinfo\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.852556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.844703    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcm4\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-kube-api-access-gtcm4\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:45.902809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:45.902724    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:46.283810 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:46.283352    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530a2744_a0ae_46dd_a4c7_ada0d33538d4.slice/crio-f6eec01cd4b26243d200ddced5f58cfd2f6215f19347d6f81cadde2867ccf460 WatchSource:0}: Error finding container f6eec01cd4b26243d200ddced5f58cfd2f6215f19347d6f81cadde2867ccf460: Status 404 returned error can't find the container with id f6eec01cd4b26243d200ddced5f58cfd2f6215f19347d6f81cadde2867ccf460
Oct 30 01:19:46.283810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.283668    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"]
Oct 30 01:19:46.287061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.287015    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:46.287197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.287113    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:46.287197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.287159    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:46.328203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.328169    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6" event={"ID":"7036635b-a1e2-4679-99dc-0e054dbbc2c5","Type":"ContainerDied","Data":"64965dc7d9f7fc4516e9397d5b9a9b957ff1dbcf536012906a12c15251839ef2"}
Oct 30 01:19:46.328203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.328189    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"
Oct 30 01:19:46.328447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.328241    5104 scope.go:122] "RemoveContainer" containerID="2aa8b19e18604d8e040a7b935d179eb349d551e210c9060eeda28e7e6b15c4d7"
Oct 30 01:19:46.330542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.330508    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" event={"ID":"530a2744-a0ae-46dd-a4c7-ada0d33538d4","Type":"ContainerStarted","Data":"f6eec01cd4b26243d200ddced5f58cfd2f6215f19347d6f81cadde2867ccf460"}
Oct 30 01:19:46.389258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.389226    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"]
Oct 30 01:19:46.395811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.395779    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e69dee-56b8f7d8ff-czkm6"]
Oct 30 01:19:46.580398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.580289    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7036635b-a1e2-4679-99dc-0e054dbbc2c5" path="/var/lib/kubelet/pods/7036635b-a1e2-4679-99dc-0e054dbbc2c5/volumes"
Oct 30 01:19:46.850168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.850068    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"]
Oct 30 01:19:46.856409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.856382    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:46.878512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.878476    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"]
Oct 30 01:19:46.881935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.881897    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-886adb-openshift-default-dockercfg-l6rhb\""
Oct 30 01:19:46.925620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.925580    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:46.941442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:46.941403    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.063981    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064135    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064288    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064318    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064502    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w2n\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-kube-api-access-x4w2n\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064641    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064685    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064785    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.074710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.064832    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169395    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w2n\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-kube-api-access-x4w2n\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169512    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169551    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169613    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169657    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169770    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169815    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169893    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.170538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.169920    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.178966    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.179621    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.180171    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.180401    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.184571    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.227273    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w2n\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-kube-api-access-x4w2n\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.236105    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.243481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.236748    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.252488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.247798    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.343008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.342966    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" event={"ID":"530a2744-a0ae-46dd-a4c7-ada0d33538d4","Type":"ContainerStarted","Data":"058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b"}
Oct 30 01:19:47.356487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.356442    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:19:47.384112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.384059    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podStartSLOduration=2.384045532 podStartE2EDuration="2.384045532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:47.382769596 +0000 UTC m=+2228.930460091" watchObservedRunningTime="2026-10-30 01:19:47.384045532 +0000 UTC m=+2228.931736027"
Oct 30 01:19:47.487951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.487861    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:47.884749 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:47.883862    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3a73a2_922c_4ecd_8e7d_4cde4f841835.slice/crio-2adcd8be0d23214bbcaeda18530e552e851390fb6e7daa506276fa403ba3d2bf WatchSource:0}: Error finding container 2adcd8be0d23214bbcaeda18530e552e851390fb6e7daa506276fa403ba3d2bf: Status 404 returned error can't find the container with id 2adcd8be0d23214bbcaeda18530e552e851390fb6e7daa506276fa403ba3d2bf
Oct 30 01:19:47.885258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.884800    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"]
Oct 30 01:19:47.887517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.887478    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:47.887642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.887578    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:47.887642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.887627    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:47.903072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.903045    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:47.910171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:47.910141    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:48.355986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:48.355945    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" event={"ID":"4f3a73a2-922c-4ecd-8e7d-4cde4f841835","Type":"ContainerStarted","Data":"6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9"}
Oct 30 01:19:48.356205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:48.356008    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" event={"ID":"4f3a73a2-922c-4ecd-8e7d-4cde4f841835","Type":"ContainerStarted","Data":"2adcd8be0d23214bbcaeda18530e552e851390fb6e7daa506276fa403ba3d2bf"}
Oct 30 01:19:48.356730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:48.356708    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:48.357837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:48.357814    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:19:48.399535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:48.399440    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podStartSLOduration=2.399425268 podStartE2EDuration="2.399425268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:48.394850952 +0000 UTC m=+2229.942541439" watchObservedRunningTime="2026-10-30 01:19:48.399425268 +0000 UTC m=+2229.947115763"
Oct 30 01:19:48.488924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:48.488309    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:48.499521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:48.497123    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:49.362804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.362766    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:49.363875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.363847    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:19:49.813800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.813765    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:19:49.885508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.885408    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-data\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.885718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.885655    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-istio-token\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-istio-token\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.886087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.886054    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-envoy\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.886710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.886665    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9798f862-937c-4699-9a5b-f35362f1458e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9798f862-937c-4699-9a5b-f35362f1458e-istio-podinfo\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.886927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.886905    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-credential-socket\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.887165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.887143    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-certs\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.887266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.887188    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-kube-api-access-vjsp8\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-kube-api-access-vjsp8\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.887266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.887225    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9798f862-937c-4699-9a5b-f35362f1458e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9798f862-937c-4699-9a5b-f35362f1458e-istiod-ca-cert\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.887381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.887294    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-socket\") pod \"9798f862-937c-4699-9a5b-f35362f1458e\" (UID: \"9798f862-937c-4699-9a5b-f35362f1458e\") "
Oct 30 01:19:49.888248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.888220    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-socket" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:49.888630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.888610    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-credential-socket" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:49.888950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.888930    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-certs" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:49.891851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.891823    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-kube-api-access-vjsp8" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "kube-api-access-vjsp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:49.892349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.892326    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9798f862-937c-4699-9a5b-f35362f1458e-istiod-ca-cert" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:49.892586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.892565    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9798f862-937c-4699-9a5b-f35362f1458e-istio-podinfo" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:49.893663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.892783    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-istio-token" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:49.893799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.893007    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-data" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:49.894692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.894666    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-envoy" pod "9798f862-937c-4699-9a5b-f35362f1458e" (UID: "9798f862-937c-4699-9a5b-f35362f1458e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990747    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990786    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990802    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990815    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9798f862-937c-4699-9a5b-f35362f1458e-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990829    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990844    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990859    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjsp8\" (UniqueName: \"kubernetes.io/projected/9798f862-937c-4699-9a5b-f35362f1458e-kube-api-access-vjsp8\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990874    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9798f862-937c-4699-9a5b-f35362f1458e-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.006489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:49.990887    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9798f862-937c-4699-9a5b-f35362f1458e-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:50.374201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.374111    5104 generic.go:356] "Generic (PLEG): container finished" podID="9798f862-937c-4699-9a5b-f35362f1458e" containerID="0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d" exitCode=0
Oct 30 01:19:50.374201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.374176    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"
Oct 30 01:19:50.374769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.374195    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" event={"ID":"9798f862-937c-4699-9a5b-f35362f1458e","Type":"ContainerDied","Data":"0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d"}
Oct 30 01:19:50.374769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.374263    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf" event={"ID":"9798f862-937c-4699-9a5b-f35362f1458e","Type":"ContainerDied","Data":"143db26276dde22c045c506871d50107838aa0ff55a7ac9e6a7f721805623e77"}
Oct 30 01:19:50.374769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.374294    5104 scope.go:122] "RemoveContainer" containerID="0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d"
Oct 30 01:19:50.387852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.387826    5104 scope.go:122] "RemoveContainer" containerID="0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d"
Oct 30 01:19:50.388222 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:19:50.388198    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d\": container with ID starting with 0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d not found: ID does not exist" containerID="0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d"
Oct 30 01:19:50.388311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.388238    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d"} err="failed to get container status \"0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d\": rpc error: code = NotFound desc = could not find container \"0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d\": container with ID starting with 0c6173e7df49850151a07163f65258b96494e1f18e89d238f05cd90dd519e08d not found: ID does not exist"
Oct 30 01:19:50.435475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.435431    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"]
Oct 30 01:19:50.447757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.447725    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-mjskf"]
Oct 30 01:19:50.580963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:50.580921    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9798f862-937c-4699-9a5b-f35362f1458e" path="/var/lib/kubelet/pods/9798f862-937c-4699-9a5b-f35362f1458e/volumes"
Oct 30 01:19:51.683239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.683207    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"]
Oct 30 01:19:51.684056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.684030    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy"
Oct 30 01:19:51.684246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.684216    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy"
Oct 30 01:19:51.684532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.684515    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9798f862-937c-4699-9a5b-f35362f1458e" containerName="istio-proxy"
Oct 30 01:19:51.690260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.690240    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:51.698027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.698000    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"]
Oct 30 01:19:51.703570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.703530    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.713073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.713042    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-e2d6d3\""
Oct 30 01:19:51.713073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.713045    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-e2d6d3-dockercfg-g4sf2\""
Oct 30 01:19:51.714823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.714794    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"]
Oct 30 01:19:51.721690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.721657    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlj7\" (UniqueName: \"kubernetes.io/projected/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-kube-api-access-hnlj7\") pod \"payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:51.722127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.722096    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:51.727732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.727696    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"]
Oct 30 01:19:51.773505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.773427    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"]
Oct 30 01:19:51.786040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.785942    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:51.802667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.802628    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"]
Oct 30 01:19:51.811475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.811361    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-add1bb\"/\"default-dockercfg-cqjm4\""
Oct 30 01:19:51.811657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.811485    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-add1bb\"/\"kube-root-ca.crt\""
Oct 30 01:19:51.811657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.811563    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-add1bb\"/\"rate-test-model-add1bb-kserve-self-signed-certs\""
Oct 30 01:19:51.811657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.811595    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-add1bb\"/\"openshift-service-ca.crt\""
Oct 30 01:19:51.830298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830246    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-model-cache\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.830501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830400    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:51.830501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830483    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caba728-a591-4de9-a400-44378f60f7c7-tls-certs\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.830657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830525    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlj7\" (UniqueName: \"kubernetes.io/projected/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-kube-api-access-hnlj7\") pod \"payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:51.830657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830584    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-tmp-dir\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.830657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830619    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c35be3-ad83-400a-9c15-07f474664c3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:51.830834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830656    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbr2\" (UniqueName: \"kubernetes.io/projected/7caba728-a591-4de9-a400-44378f60f7c7-kube-api-access-7sbr2\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.830834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830687    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-home\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.830834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830782    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp82\" (UniqueName: \"kubernetes.io/projected/60c35be3-ad83-400a-9c15-07f474664c3c-kube-api-access-zfp82\") pod \"payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:51.830992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.830909    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-dshm\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.831657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.831629    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:51.850450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.850418    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlj7\" (UniqueName: \"kubernetes.io/projected/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-kube-api-access-hnlj7\") pod \"payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:51.933548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933442    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-tmp-dir\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.933548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933509    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c35be3-ad83-400a-9c15-07f474664c3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:51.933548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933545    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbr2\" (UniqueName: \"kubernetes.io/projected/7caba728-a591-4de9-a400-44378f60f7c7-kube-api-access-7sbr2\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.933859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933567    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-home\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.933859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933653    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp82\" (UniqueName: \"kubernetes.io/projected/60c35be3-ad83-400a-9c15-07f474664c3c-kube-api-access-zfp82\") pod \"payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:51.933859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933763    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-dshm\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.933859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933839    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-model-cache\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.934086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933927    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-tmp-dir\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.934086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.933932    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caba728-a591-4de9-a400-44378f60f7c7-tls-certs\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.934475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.934437    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-home\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.934638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.934508    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-model-cache\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.934638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.934524    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c35be3-ad83-400a-9c15-07f474664c3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:51.936582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.936558    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-dshm\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.936699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.936635    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caba728-a591-4de9-a400-44378f60f7c7-tls-certs\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.953319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.953280    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbr2\" (UniqueName: \"kubernetes.io/projected/7caba728-a591-4de9-a400-44378f60f7c7-kube-api-access-7sbr2\") pod \"rate-test-model-add1bb-kserve-6664bd5965-hq695\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") " pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:51.953546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:51.953348    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp82\" (UniqueName: \"kubernetes.io/projected/60c35be3-ad83-400a-9c15-07f474664c3c-kube-api-access-zfp82\") pod \"payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:52.010099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.010061    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:52.021882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.021849    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:52.135840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.135791    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:52.377479 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:52.371488    5104 logging.go:55] [core] [Channel #9181 SubChannel #9182]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.234:50052", ServerName: "10.132.0.234:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.234:50052: connect: connection refused"
Oct 30 01:19:52.397083 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:52.389254    5104 logging.go:55] [core] [Channel #9183 SubChannel #9184]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.235:50052", ServerName: "10.132.0.235:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.235:50052: connect: connection refused"
Oct 30 01:19:52.501825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.501780    5104 generic.go:356] "Generic (PLEG): container finished" podID="ca181706-ab97-4fec-bc71-6b55126060f3" containerID="d64b4f7afd768ca9cb2f0e90196fe2ea81c2371e6b7c629c6a43b6ced6b1ade7" exitCode=0
Oct 30 01:19:52.501991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.501926    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" event={"ID":"ca181706-ab97-4fec-bc71-6b55126060f3","Type":"ContainerDied","Data":"d64b4f7afd768ca9cb2f0e90196fe2ea81c2371e6b7c629c6a43b6ced6b1ade7"}
Oct 30 01:19:52.504133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.504062    5104 generic.go:356] "Generic (PLEG): container finished" podID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerID="6b7f6afa7887eba8c3cabb4e85d1195ef4b73f54f8860d16fb48008116715d0b" exitCode=0
Oct 30 01:19:52.504133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.504127    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" event={"ID":"f7636dbe-33ba-45d3-a6a8-884db1a2c880","Type":"ContainerDied","Data":"6b7f6afa7887eba8c3cabb4e85d1195ef4b73f54f8860d16fb48008116715d0b"}
Oct 30 01:19:52.561168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.561135    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"]
Oct 30 01:19:52.561361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.561323    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy" containerID="cri-o://9af10a350511a482bf7c4d81c485965fd1be8c367bffdf4d20134e9a6fcb957c" gracePeriod=30
Oct 30 01:19:52.880491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.862653    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:19:52.913222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.911526    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"]
Oct 30 01:19:52.919628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.919595    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"]
Oct 30 01:19:52.938799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.938758    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:19:52.970973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.970938    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca181706-ab97-4fec-bc71-6b55126060f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca181706-ab97-4fec-bc71-6b55126060f3-plugins-config-volume\") pod \"ca181706-ab97-4fec-bc71-6b55126060f3\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") "
Oct 30 01:19:52.971150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.970990    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7636dbe-33ba-45d3-a6a8-884db1a2c880-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7636dbe-33ba-45d3-a6a8-884db1a2c880-plugins-config-volume\") pod \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") "
Oct 30 01:19:52.971150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.971126    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca181706-ab97-4fec-bc71-6b55126060f3-kube-api-access-zwc62\" (UniqueName: \"kubernetes.io/projected/ca181706-ab97-4fec-bc71-6b55126060f3-kube-api-access-zwc62\") pod \"ca181706-ab97-4fec-bc71-6b55126060f3\" (UID: \"ca181706-ab97-4fec-bc71-6b55126060f3\") "
Oct 30 01:19:52.971271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.971237    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7636dbe-33ba-45d3-a6a8-884db1a2c880-kube-api-access-c7pcp\" (UniqueName: \"kubernetes.io/projected/f7636dbe-33ba-45d3-a6a8-884db1a2c880-kube-api-access-c7pcp\") pod \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\" (UID: \"f7636dbe-33ba-45d3-a6a8-884db1a2c880\") "
Oct 30 01:19:52.972767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.972739    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7636dbe-33ba-45d3-a6a8-884db1a2c880-plugins-config-volume" pod "f7636dbe-33ba-45d3-a6a8-884db1a2c880" (UID: "f7636dbe-33ba-45d3-a6a8-884db1a2c880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:52.973161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.973141    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca181706-ab97-4fec-bc71-6b55126060f3-plugins-config-volume" pod "ca181706-ab97-4fec-bc71-6b55126060f3" (UID: "ca181706-ab97-4fec-bc71-6b55126060f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:52.974088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.974057    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7636dbe-33ba-45d3-a6a8-884db1a2c880-kube-api-access-c7pcp" pod "f7636dbe-33ba-45d3-a6a8-884db1a2c880" (UID: "f7636dbe-33ba-45d3-a6a8-884db1a2c880"). InnerVolumeSpecName "kube-api-access-c7pcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:52.975148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:52.975121    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca181706-ab97-4fec-bc71-6b55126060f3-kube-api-access-zwc62" pod "ca181706-ab97-4fec-bc71-6b55126060f3" (UID: "ca181706-ab97-4fec-bc71-6b55126060f3"). InnerVolumeSpecName "kube-api-access-zwc62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:53.073980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.073891    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca181706-ab97-4fec-bc71-6b55126060f3-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.073980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.073923    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7636dbe-33ba-45d3-a6a8-884db1a2c880-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.073980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.073933    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwc62\" (UniqueName: \"kubernetes.io/projected/ca181706-ab97-4fec-bc71-6b55126060f3-kube-api-access-zwc62\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.073980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.073942    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pcp\" (UniqueName: \"kubernetes.io/projected/f7636dbe-33ba-45d3-a6a8-884db1a2c880-kube-api-access-c7pcp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.140721 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:53.140681    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c35be3_ad83_400a_9c15_07f474664c3c.slice/crio-892c1443612a96a1982a18a93f1e4ad86039bae617b6ca4f250628b1a5449680 WatchSource:0}: Error finding container 892c1443612a96a1982a18a93f1e4ad86039bae617b6ca4f250628b1a5449680: Status 404 returned error can't find the container with id 892c1443612a96a1982a18a93f1e4ad86039bae617b6ca4f250628b1a5449680
Oct 30 01:19:53.140894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.140761    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"]
Oct 30 01:19:53.372196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.372103    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.234:50052\" within 1s: context deadline exceeded"
Oct 30 01:19:53.372196 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:53.372143    5104 logging.go:55] [core] [Channel #9181 SubChannel #9182]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.234:50052", ServerName: "10.132.0.234:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.234:50052: operation was canceled"
Oct 30 01:19:53.372196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.372193    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:19:53.373076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.373046    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.235:50052\" within 1s: context deadline exceeded"
Oct 30 01:19:53.373142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.373129    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:19:53.516250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.516203    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk" event={"ID":"60c35be3-ad83-400a-9c15-07f474664c3c","Type":"ContainerStarted","Data":"4cbd8c349168d3b9db750c1cffb7009824907c32cfa865decc67a242c73a8848"}
Oct 30 01:19:53.516431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.516274    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk" event={"ID":"60c35be3-ad83-400a-9c15-07f474664c3c","Type":"ContainerStarted","Data":"892c1443612a96a1982a18a93f1e4ad86039bae617b6ca4f250628b1a5449680"}
Oct 30 01:19:53.516431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.516303    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:19:53.518238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.518206    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" event={"ID":"7caba728-a591-4de9-a400-44378f60f7c7","Type":"ContainerStarted","Data":"298696350ecebe95e009b91b958600ab85c621d2e5e3575258e6221fdfab4fa4"}
Oct 30 01:19:53.518390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.518249    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" event={"ID":"7caba728-a591-4de9-a400-44378f60f7c7","Type":"ContainerStarted","Data":"b8123f12364763fc3211d53cfa96418ff5a4c496a6daacfb82cd69546385efed"}
Oct 30 01:19:53.519848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.519826    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"
Oct 30 01:19:53.519979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.519874    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x" event={"ID":"ca181706-ab97-4fec-bc71-6b55126060f3","Type":"ContainerDied","Data":"a04158a53114cdecd0e3a2114f50e0e972e6c3f9f15ba64577b7b32b81256ad4"}
Oct 30 01:19:53.519979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.519930    5104 scope.go:122] "RemoveContainer" containerID="d64b4f7afd768ca9cb2f0e90196fe2ea81c2371e6b7c629c6a43b6ced6b1ade7"
Oct 30 01:19:53.522760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.522731    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt" event={"ID":"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d","Type":"ContainerStarted","Data":"ae8945706034fa659b09af0482fea480534c276d44f5cf7f8e5c3197623f7b75"}
Oct 30 01:19:53.522865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.522784    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt" event={"ID":"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d","Type":"ContainerStarted","Data":"78810ba3457dcaffcf79a6258268ca1221cb53b417c59bcad2db9cea18cfdb12"}
Oct 30 01:19:53.523542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.523519    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:19:53.525316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.525292    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"
Oct 30 01:19:53.525401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.525290    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf" event={"ID":"f7636dbe-33ba-45d3-a6a8-884db1a2c880","Type":"ContainerDied","Data":"4bbe42fcdd3c77f4cb972c4f2fd0e66792be81863c9c2c08fa02b70602c4810d"}
Oct 30 01:19:53.537243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.537208    5104 scope.go:122] "RemoveContainer" containerID="6b7f6afa7887eba8c3cabb4e85d1195ef4b73f54f8860d16fb48008116715d0b"
Oct 30 01:19:53.554801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.554744    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk" podStartSLOduration=2.554723946 podStartE2EDuration="2.554723946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:53.551798626 +0000 UTC m=+2235.099489122" watchObservedRunningTime="2026-10-30 01:19:53.554723946 +0000 UTC m=+2235.102414441"
Oct 30 01:19:53.591808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.591749    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt" podStartSLOduration=2.591733867 podStartE2EDuration="2.591733867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:53.587714816 +0000 UTC m=+2235.135405293" watchObservedRunningTime="2026-10-30 01:19:53.591733867 +0000 UTC m=+2235.139424362"
Oct 30 01:19:53.627694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.627598    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"]
Oct 30 01:19:53.634790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.634752    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e5711-786884fc6d-hzk2x"]
Oct 30 01:19:53.664270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.664208    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podStartSLOduration=2.664189491 podStartE2EDuration="2.664189491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:53.663052629 +0000 UTC m=+2235.210743127" watchObservedRunningTime="2026-10-30 01:19:53.664189491 +0000 UTC m=+2235.211879989"
Oct 30 01:19:53.705618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.705578    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"]
Oct 30 01:19:53.712034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.711999    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e5711-66d89f5979-kd7mf"]
Oct 30 01:19:53.778241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.778201    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"]
Oct 30 01:19:53.779405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.779361    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerName="payload-pre-processing"
Oct 30 01:19:53.780409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.780384    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerName="payload-pre-processing"
Oct 30 01:19:53.780671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.780654    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" containerName="payload-processing"
Oct 30 01:19:53.780771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.780753    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" containerName="payload-processing"
Oct 30 01:19:53.781006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.780989    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" containerName="payload-pre-processing"
Oct 30 01:19:53.781085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.781022    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" containerName="payload-processing"
Oct 30 01:19:53.787284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.787260    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.806158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.806108    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-d81f0a-openshift-default-dockercfg-447gk\""
Oct 30 01:19:53.813509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.813475    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"]
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956236    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-certs\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956335    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-podinfo\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956368    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xph48\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-kube-api-access-xph48\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956397    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istiod-ca-cert\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956437    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-data\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956514    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-envoy\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956580    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-token\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956628    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-credential-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:53.964479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:53.956761    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.022693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.022652    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:19:54.094375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.058914    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.105484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.095049    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-certs\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.125637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.125597    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-podinfo\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.126402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.126377    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xph48\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-kube-api-access-xph48\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.126929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.126903    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istiod-ca-cert\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.127927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.127906    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-data\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.130339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.130319    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-envoy\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.130941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.130919    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-token\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.131404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.131383    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-credential-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.131784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.130214    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-data\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.132973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.059328    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.133113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.127822    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istiod-ca-cert\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.133218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.106071    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-certs\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.133312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.132225    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-credential-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.136775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.136752    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-envoy\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.147420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.147386    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph48\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-kube-api-access-xph48\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.154962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.152119    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-podinfo\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.154962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.154835    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-token\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.407292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.407199    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:54.452203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.452161    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"]
Oct 30 01:19:54.458323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.458294    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.497708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.497247    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-886adb-dockercfg-2pdt8\""
Oct 30 01:19:54.498574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.498322    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-886adb\""
Oct 30 01:19:54.579560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.568690    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95743e3-96fd-481b-99e6-88257ec2d360-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.579560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.568759    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvk6\" (UniqueName: \"kubernetes.io/projected/d95743e3-96fd-481b-99e6-88257ec2d360-kube-api-access-lqvk6\") pod \"payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.683481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.680756    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95743e3-96fd-481b-99e6-88257ec2d360-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.683481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.680831    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvk6\" (UniqueName: \"kubernetes.io/projected/d95743e3-96fd-481b-99e6-88257ec2d360-kube-api-access-lqvk6\") pod \"payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.751499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.751427    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95743e3-96fd-481b-99e6-88257ec2d360-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.833483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.833271    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvk6\" (UniqueName: \"kubernetes.io/projected/d95743e3-96fd-481b-99e6-88257ec2d360-kube-api-access-lqvk6\") pod \"payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.862310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.862271    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca181706-ab97-4fec-bc71-6b55126060f3" path="/var/lib/kubelet/pods/ca181706-ab97-4fec-bc71-6b55126060f3/volumes"
Oct 30 01:19:54.863296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.863266    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7636dbe-33ba-45d3-a6a8-884db1a2c880" path="/var/lib/kubelet/pods/f7636dbe-33ba-45d3-a6a8-884db1a2c880/volumes"
Oct 30 01:19:54.863995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.863961    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"]
Oct 30 01:19:54.864225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.864201    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"]
Oct 30 01:19:54.882293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.882262    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"]
Oct 30 01:19:54.883887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.882304    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:54.938070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.937987    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:54.938304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.938278    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt9qm\" (UniqueName: \"kubernetes.io/projected/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-kube-api-access-dt9qm\") pod \"payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:54.958207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.958171    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:54.959772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.959744    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"]
Oct 30 01:19:54.960193 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:54.960169    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d563dff_75e3_48e2_b6ca_4dd41a6cca56.slice/crio-68f97f90abbbe74aee8dc4737d04d5748974474d35a9537e87645071c3dc0ef0 WatchSource:0}: Error finding container 68f97f90abbbe74aee8dc4737d04d5748974474d35a9537e87645071c3dc0ef0: Status 404 returned error can't find the container with id 68f97f90abbbe74aee8dc4737d04d5748974474d35a9537e87645071c3dc0ef0
Oct 30 01:19:54.977579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.977530    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:54.977927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.977639    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:54.977927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:54.977683    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:19:55.040359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.039972    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:55.040359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.040066    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt9qm\" (UniqueName: \"kubernetes.io/projected/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-kube-api-access-dt9qm\") pod \"payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:55.041081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.041054    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:55.062863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.060684    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt9qm\" (UniqueName: \"kubernetes.io/projected/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-kube-api-access-dt9qm\") pod \"payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:55.217540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.217412    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:55.286382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.286281    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"]
Oct 30 01:19:55.544404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.544377    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"]
Oct 30 01:19:55.546047 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:19:55.546003    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc11c49_e9b1_4649_a0cf_2917bbff63b7.slice/crio-61435ee4f952ee2d1aa723ece5157cd7a20c2fad4258be3287eca08d5c7837ae WatchSource:0}: Error finding container 61435ee4f952ee2d1aa723ece5157cd7a20c2fad4258be3287eca08d5c7837ae: Status 404 returned error can't find the container with id 61435ee4f952ee2d1aa723ece5157cd7a20c2fad4258be3287eca08d5c7837ae
Oct 30 01:19:55.764440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.764346    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" event={"ID":"d95743e3-96fd-481b-99e6-88257ec2d360","Type":"ContainerStarted","Data":"eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78"}
Oct 30 01:19:55.764440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.764412    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" event={"ID":"d95743e3-96fd-481b-99e6-88257ec2d360","Type":"ContainerStarted","Data":"09a844023c173a40c2178be5f553d1f313150171eb37e6b17e56e1c21eeec9cc"}
Oct 30 01:19:55.764440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.764437    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:19:55.766330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.766302    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" event={"ID":"4fc11c49-e9b1-4649-a0cf-2917bbff63b7","Type":"ContainerStarted","Data":"098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860"}
Oct 30 01:19:55.766503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.766346    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" event={"ID":"4fc11c49-e9b1-4649-a0cf-2917bbff63b7","Type":"ContainerStarted","Data":"61435ee4f952ee2d1aa723ece5157cd7a20c2fad4258be3287eca08d5c7837ae"}
Oct 30 01:19:55.766503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.766368    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:19:55.768093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.768068    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" event={"ID":"1d563dff-75e3-48e2-b6ca-4dd41a6cca56","Type":"ContainerStarted","Data":"f3631caf395c09e7e20feaf56189b9c3db723ec3bfe09a8007ebcc93e2ce279f"}
Oct 30 01:19:55.768177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.768115    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" event={"ID":"1d563dff-75e3-48e2-b6ca-4dd41a6cca56","Type":"ContainerStarted","Data":"68f97f90abbbe74aee8dc4737d04d5748974474d35a9537e87645071c3dc0ef0"}
Oct 30 01:19:55.804400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.804332    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" podStartSLOduration=1.8043159050000002 podStartE2EDuration="1.804315905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:55.799732042 +0000 UTC m=+2237.347422538" watchObservedRunningTime="2026-10-30 01:19:55.804315905 +0000 UTC m=+2237.352006400"
Oct 30 01:19:55.836139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.836087    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" podStartSLOduration=1.836072009 podStartE2EDuration="1.836072009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:55.834807447 +0000 UTC m=+2237.382497942" watchObservedRunningTime="2026-10-30 01:19:55.836072009 +0000 UTC m=+2237.383762536"
Oct 30 01:19:55.899825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:55.899767    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podStartSLOduration=2.899751377 podStartE2EDuration="2.899751377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:55.885850186 +0000 UTC m=+2237.433540683" watchObservedRunningTime="2026-10-30 01:19:55.899751377 +0000 UTC m=+2237.447441873"
Oct 30 01:19:56.408137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:56.408099    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:56.415233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:56.415201    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:56.777099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:56.777063    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:56.778104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:56.778074    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:19:57.805669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.805623    5104 generic.go:356] "Generic (PLEG): container finished" podID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerID="9af10a350511a482bf7c4d81c485965fd1be8c367bffdf4d20134e9a6fcb957c" exitCode=0
Oct 30 01:19:57.806178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.805700    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" event={"ID":"3716857a-a939-45b8-a24e-8f30afbc35ba","Type":"ContainerDied","Data":"9af10a350511a482bf7c4d81c485965fd1be8c367bffdf4d20134e9a6fcb957c"}
Oct 30 01:19:57.806178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.805786    5104 scope.go:122] "RemoveContainer" containerID="c2ed956a6225d43ddff435e465f46c8a9514e4583f58c44453833388050ed3a0"
Oct 30 01:19:57.885817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.885787    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:19:57.945743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.945699    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-token\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-token\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.945941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.945800    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-socket\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.945941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.945837    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3716857a-a939-45b8-a24e-8f30afbc35ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3716857a-a939-45b8-a24e-8f30afbc35ba-istiod-ca-cert\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.945941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.945920    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-podinfo\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.946088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.945992    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-certs\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.946127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.946083    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-kube-api-access-rkqq5\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-kube-api-access-rkqq5\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.946212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.946188    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-data\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.946282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.946235    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-envoy\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.946400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.946381    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-credential-socket\") pod \"3716857a-a939-45b8-a24e-8f30afbc35ba\" (UID: \"3716857a-a939-45b8-a24e-8f30afbc35ba\") "
Oct 30 01:19:57.947381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.947348    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-credential-socket" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.947655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.947630    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-certs" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.948980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.948126    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-socket" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.948980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.948624    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-data" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:57.948980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.948801    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3716857a-a939-45b8-a24e-8f30afbc35ba-istiod-ca-cert" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:57.948980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.948944    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-token" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:57.950178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.950150    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-kube-api-access-rkqq5" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "kube-api-access-rkqq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:57.950818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.950790    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-podinfo" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:57.953878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:57.953853    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-envoy" pod "3716857a-a939-45b8-a24e-8f30afbc35ba" (UID: "3716857a-a939-45b8-a24e-8f30afbc35ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:58.049143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049103    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkqq5\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-kube-api-access-rkqq5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049141    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049152    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049160    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049168    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049178    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049187    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3716857a-a939-45b8-a24e-8f30afbc35ba-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049196    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3716857a-a939-45b8-a24e-8f30afbc35ba-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.049397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.049204    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3716857a-a939-45b8-a24e-8f30afbc35ba-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.819088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.819039    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"
Oct 30 01:19:58.819653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.819079    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz" event={"ID":"3716857a-a939-45b8-a24e-8f30afbc35ba","Type":"ContainerDied","Data":"febc991364315b9242ce84cc1ace982b62c2369e3e8008bd95c581c4d4518158"}
Oct 30 01:19:58.819653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.819149    5104 scope.go:122] "RemoveContainer" containerID="9af10a350511a482bf7c4d81c485965fd1be8c367bffdf4d20134e9a6fcb957c"
Oct 30 01:19:58.872177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.872143    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"]
Oct 30 01:19:58.881383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:19:58.881346    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-n2lkz"]
Oct 30 01:20:00.174333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.174291    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"]
Oct 30 01:20:00.176913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.175133    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy"
Oct 30 01:20:00.176913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.175150    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy"
Oct 30 01:20:00.176913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.175163    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy"
Oct 30 01:20:00.176913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.175190    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy"
Oct 30 01:20:00.176913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.175352    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy"
Oct 30 01:20:00.176913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.175363    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" containerName="istio-proxy"
Oct 30 01:20:00.179823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.179794    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.197866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.197822    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"]
Oct 30 01:20:00.202048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.202021    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-d81f0a\""
Oct 30 01:20:00.202261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.202080    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-d81f0a-dockercfg-8rpnv\""
Oct 30 01:20:00.339487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.323268    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4352416c-a1ef-417c-9052-6274a29fd052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.339487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.323347    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrdz\" (UniqueName: \"kubernetes.io/projected/4352416c-a1ef-417c-9052-6274a29fd052-kube-api-access-mbrdz\") pod \"payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.466695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.466609    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4352416c-a1ef-417c-9052-6274a29fd052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.467022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.466984    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4352416c-a1ef-417c-9052-6274a29fd052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.479919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.479881    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"]
Oct 30 01:20:00.480631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.480596    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrdz\" (UniqueName: \"kubernetes.io/projected/4352416c-a1ef-417c-9052-6274a29fd052-kube-api-access-mbrdz\") pod \"payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.512412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.512379    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrdz\" (UniqueName: \"kubernetes.io/projected/4352416c-a1ef-417c-9052-6274a29fd052-kube-api-access-mbrdz\") pod \"payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.513170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.513143    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:00.526419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.526380    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"]
Oct 30 01:20:00.580559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.580524    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3716857a-a939-45b8-a24e-8f30afbc35ba" path="/var/lib/kubelet/pods/3716857a-a939-45b8-a24e-8f30afbc35ba/volumes"
Oct 30 01:20:00.683666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.683619    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6d6\" (UniqueName: \"kubernetes.io/projected/c24262f6-4765-4404-86c0-87cbcd13dd97-kube-api-access-tb6d6\") pod \"payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:00.683885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.683810    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24262f6-4765-4404-86c0-87cbcd13dd97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:00.785247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.785147    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6d6\" (UniqueName: \"kubernetes.io/projected/c24262f6-4765-4404-86c0-87cbcd13dd97-kube-api-access-tb6d6\") pod \"payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:00.785247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.785242    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24262f6-4765-4404-86c0-87cbcd13dd97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:00.786090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.786060    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24262f6-4765-4404-86c0-87cbcd13dd97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:00.803223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.803189    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:00.803576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.803551    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6d6\" (UniqueName: \"kubernetes.io/projected/c24262f6-4765-4404-86c0-87cbcd13dd97-kube-api-access-tb6d6\") pod \"payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:00.856855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:00.856820    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:01.111082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.111046    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"]
Oct 30 01:20:01.113546 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:01.113517    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4352416c_a1ef_417c_9052_6274a29fd052.slice/crio-9d2f0dfff4a3ff71d73b03b09d7974124d2fd82fbb2f7e63a9433f7987118b76 WatchSource:0}: Error finding container 9d2f0dfff4a3ff71d73b03b09d7974124d2fd82fbb2f7e63a9433f7987118b76: Status 404 returned error can't find the container with id 9d2f0dfff4a3ff71d73b03b09d7974124d2fd82fbb2f7e63a9433f7987118b76
Oct 30 01:20:01.246447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.246415    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"]
Oct 30 01:20:01.246840 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:01.246753    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24262f6_4765_4404_86c0_87cbcd13dd97.slice/crio-529aef5988073c820203315fc36a23efecb09542fefb7d0435ee8e82380fb20d WatchSource:0}: Error finding container 529aef5988073c820203315fc36a23efecb09542fefb7d0435ee8e82380fb20d: Status 404 returned error can't find the container with id 529aef5988073c820203315fc36a23efecb09542fefb7d0435ee8e82380fb20d
Oct 30 01:20:01.853004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.852966    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" event={"ID":"c24262f6-4765-4404-86c0-87cbcd13dd97","Type":"ContainerStarted","Data":"768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae"}
Oct 30 01:20:01.853193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.853034    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" event={"ID":"c24262f6-4765-4404-86c0-87cbcd13dd97","Type":"ContainerStarted","Data":"529aef5988073c820203315fc36a23efecb09542fefb7d0435ee8e82380fb20d"}
Oct 30 01:20:01.853193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.853067    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:01.855088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.855058    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" event={"ID":"4352416c-a1ef-417c-9052-6274a29fd052","Type":"ContainerStarted","Data":"43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d"}
Oct 30 01:20:01.855256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.855115    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:01.855256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.855137    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" event={"ID":"4352416c-a1ef-417c-9052-6274a29fd052","Type":"ContainerStarted","Data":"9d2f0dfff4a3ff71d73b03b09d7974124d2fd82fbb2f7e63a9433f7987118b76"}
Oct 30 01:20:01.891875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.891809    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" podStartSLOduration=1.89179272 podStartE2EDuration="1.89179272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:01.888632381 +0000 UTC m=+2243.436322876" watchObservedRunningTime="2026-10-30 01:20:01.89179272 +0000 UTC m=+2243.439483251"
Oct 30 01:20:01.928245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:01.928193    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" podStartSLOduration=1.928177721 podStartE2EDuration="1.928177721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:01.924865428 +0000 UTC m=+2243.472555924" watchObservedRunningTime="2026-10-30 01:20:01.928177721 +0000 UTC m=+2243.475868217"
Oct 30 01:20:02.022551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:02.022509    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:20:02.046492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:02.037797    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:20:02.878738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:02.878703    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:20:04.760521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:04.760449    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:20:05.006852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:05.006809    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:20:05.771641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:05.771607    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:20:06.859479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:06.855658    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:20:06.878806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:06.878770    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:20:09.106403 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:20:09.106363    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:20:12.867749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:12.867715    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:20:12.868197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:12.868176    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:20:13.336402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.336368    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"]
Oct 30 01:20:13.342518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.342490    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.356306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.356271    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:13.356306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.356301    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:13.359446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.359417    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"]
Oct 30 01:20:13.364343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.364316    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:13.364583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.364354    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:20:13.461359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.461323    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9xc\" (UniqueName: \"kubernetes.io/projected/be997695-27a5-4279-96a0-35cac451f1f8-kube-api-access-sx9xc\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.461565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.461395    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.461622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.461568    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.563771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.563732    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9xc\" (UniqueName: \"kubernetes.io/projected/be997695-27a5-4279-96a0-35cac451f1f8-kube-api-access-sx9xc\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.563969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.563839    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.564030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.563981    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.564720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.564695    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.566536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.566500    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.582533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.582492    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9xc\" (UniqueName: \"kubernetes.io/projected/be997695-27a5-4279-96a0-35cac451f1f8-kube-api-access-sx9xc\") pod \"maas-api-revoke-sub-be9d0a7b-ssx5j\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.663695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.663600    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:13.982888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:13.982858    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"]
Oct 30 01:20:14.159527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.159491    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j" event={"ID":"be997695-27a5-4279-96a0-35cac451f1f8","Type":"ContainerStarted","Data":"ca4326e7b19ec0de16ccad13dbdab992cf89357d10ad838a35cef5438aef0183"}
Oct 30 01:20:14.729073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.729034    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"]
Oct 30 01:20:14.734056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.734028    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.753038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.752585    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"]
Oct 30 01:20:14.890410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.890361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.890639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.890535    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shllb\" (UniqueName: \"kubernetes.io/projected/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-kube-api-access-shllb\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.890747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.890727    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.992790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.992697    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.992790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.992769    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shllb\" (UniqueName: \"kubernetes.io/projected/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-kube-api-access-shllb\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.993334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.992917    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.993570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.993545    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:14.996141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:14.996117    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:15.011658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:15.011625    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shllb\" (UniqueName: \"kubernetes.io/projected/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-kube-api-access-shllb\") pod \"maas-api-revoke-sub-5f85cbc6-jlcx8\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:15.050255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:15.050214    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:15.196249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:15.196211    5104 generic.go:356] "Generic (PLEG): container finished" podID="be997695-27a5-4279-96a0-35cac451f1f8" containerID="74c4e9de296b0ce5d7cf9cf75da7058ab3d2048ee8992f8699d3679d32e34e9f" exitCode=0
Oct 30 01:20:15.196610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:15.196584    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j" event={"ID":"be997695-27a5-4279-96a0-35cac451f1f8","Type":"ContainerDied","Data":"74c4e9de296b0ce5d7cf9cf75da7058ab3d2048ee8992f8699d3679d32e34e9f"}
Oct 30 01:20:15.365162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:15.365122    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"]
Oct 30 01:20:15.365603 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:15.365572    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee055e72_2e4b_4e4f_82c4_9649c6dffb11.slice/crio-fbaa736cb56b07ea5b2dcf0b9d496accd1657dda30d45c453c7992f6fe64bbe9 WatchSource:0}: Error finding container fbaa736cb56b07ea5b2dcf0b9d496accd1657dda30d45c453c7992f6fe64bbe9: Status 404 returned error can't find the container with id fbaa736cb56b07ea5b2dcf0b9d496accd1657dda30d45c453c7992f6fe64bbe9
Oct 30 01:20:16.209967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.209927    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8" event={"ID":"ee055e72-2e4b-4e4f-82c4-9649c6dffb11","Type":"ContainerStarted","Data":"b05a34828ccb7d39ffcbee11e29ad212d4161cba93e10c7edc91b90118eb4654"}
Oct 30 01:20:16.210438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.209996    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8" event={"ID":"ee055e72-2e4b-4e4f-82c4-9649c6dffb11","Type":"ContainerStarted","Data":"fbaa736cb56b07ea5b2dcf0b9d496accd1657dda30d45c453c7992f6fe64bbe9"}
Oct 30 01:20:16.251552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.251477    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8" podStartSLOduration=2.251441928 podStartE2EDuration="2.251441928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:16.245330881 +0000 UTC m=+2257.793021380" watchObservedRunningTime="2026-10-30 01:20:16.251441928 +0000 UTC m=+2257.799132425"
Oct 30 01:20:16.408985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.408951    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:16.521813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.521708    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-xks-ca\") pod \"be997695-27a5-4279-96a0-35cac451f1f8\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") "
Oct 30 01:20:16.522009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.521823    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-openshift-ca\") pod \"be997695-27a5-4279-96a0-35cac451f1f8\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") "
Oct 30 01:20:16.522009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.521881    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be997695-27a5-4279-96a0-35cac451f1f8-kube-api-access-sx9xc\" (UniqueName: \"kubernetes.io/projected/be997695-27a5-4279-96a0-35cac451f1f8-kube-api-access-sx9xc\") pod \"be997695-27a5-4279-96a0-35cac451f1f8\" (UID: \"be997695-27a5-4279-96a0-35cac451f1f8\") "
Oct 30 01:20:16.522268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.522233    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-openshift-ca" pod "be997695-27a5-4279-96a0-35cac451f1f8" (UID: "be997695-27a5-4279-96a0-35cac451f1f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:16.522588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.522566    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:16.524101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.524071    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be997695-27a5-4279-96a0-35cac451f1f8-kube-api-access-sx9xc" pod "be997695-27a5-4279-96a0-35cac451f1f8" (UID: "be997695-27a5-4279-96a0-35cac451f1f8"). InnerVolumeSpecName "kube-api-access-sx9xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:16.524471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.524440    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-xks-ca" pod "be997695-27a5-4279-96a0-35cac451f1f8" (UID: "be997695-27a5-4279-96a0-35cac451f1f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:16.623953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.623913    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9xc\" (UniqueName: \"kubernetes.io/projected/be997695-27a5-4279-96a0-35cac451f1f8-kube-api-access-sx9xc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:16.623953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:16.623946    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be997695-27a5-4279-96a0-35cac451f1f8-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:17.221630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:17.221595    5104 generic.go:356] "Generic (PLEG): container finished" podID="ee055e72-2e4b-4e4f-82c4-9649c6dffb11" containerID="b05a34828ccb7d39ffcbee11e29ad212d4161cba93e10c7edc91b90118eb4654" exitCode=0
Oct 30 01:20:17.222084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:17.221680    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8" event={"ID":"ee055e72-2e4b-4e4f-82c4-9649c6dffb11","Type":"ContainerDied","Data":"b05a34828ccb7d39ffcbee11e29ad212d4161cba93e10c7edc91b90118eb4654"}
Oct 30 01:20:17.223593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:17.223570    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"
Oct 30 01:20:17.223746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:17.223568    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j" event={"ID":"be997695-27a5-4279-96a0-35cac451f1f8","Type":"ContainerDied","Data":"ca4326e7b19ec0de16ccad13dbdab992cf89357d10ad838a35cef5438aef0183"}
Oct 30 01:20:17.223746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:17.223689    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca4326e7b19ec0de16ccad13dbdab992cf89357d10ad838a35cef5438aef0183"
Oct 30 01:20:18.447344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.447313    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:18.541537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.522973    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-openshift-ca\") pod \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") "
Oct 30 01:20:18.541537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.523480    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-kube-api-access-shllb\" (UniqueName: \"kubernetes.io/projected/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-kube-api-access-shllb\") pod \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") "
Oct 30 01:20:18.541537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.523582    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-xks-ca\") pod \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\" (UID: \"ee055e72-2e4b-4e4f-82c4-9649c6dffb11\") "
Oct 30 01:20:18.563482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.554068    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-openshift-ca" pod "ee055e72-2e4b-4e4f-82c4-9649c6dffb11" (UID: "ee055e72-2e4b-4e4f-82c4-9649c6dffb11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:18.580125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.580070    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-kube-api-access-shllb" pod "ee055e72-2e4b-4e4f-82c4-9649c6dffb11" (UID: "ee055e72-2e4b-4e4f-82c4-9649c6dffb11"). InnerVolumeSpecName "kube-api-access-shllb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:18.594975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.594944    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-xks-ca" pod "ee055e72-2e4b-4e4f-82c4-9649c6dffb11" (UID: "ee055e72-2e4b-4e4f-82c4-9649c6dffb11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:18.629551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.629513    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shllb\" (UniqueName: \"kubernetes.io/projected/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-kube-api-access-shllb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:18.629551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.629551    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:18.629820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:18.629565    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee055e72-2e4b-4e4f-82c4-9649c6dffb11-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:19.244310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:19.244275    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"
Oct 30 01:20:19.244517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:19.244312    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8" event={"ID":"ee055e72-2e4b-4e4f-82c4-9649c6dffb11","Type":"ContainerDied","Data":"fbaa736cb56b07ea5b2dcf0b9d496accd1657dda30d45c453c7992f6fe64bbe9"}
Oct 30 01:20:19.244517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:19.244404    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbaa736cb56b07ea5b2dcf0b9d496accd1657dda30d45c453c7992f6fe64bbe9"
Oct 30 01:20:23.671735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.671691    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"]
Oct 30 01:20:23.672510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.672491    5104 cpu_manager.go:414] "containerMap: removing container" podUID="be997695-27a5-4279-96a0-35cac451f1f8" containerName="revoke-keys"
Oct 30 01:20:23.672510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.672510    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be997695-27a5-4279-96a0-35cac451f1f8" containerName="revoke-keys"
Oct 30 01:20:23.672599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.672541    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ee055e72-2e4b-4e4f-82c4-9649c6dffb11" containerName="revoke-keys"
Oct 30 01:20:23.672599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.672546    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee055e72-2e4b-4e4f-82c4-9649c6dffb11" containerName="revoke-keys"
Oct 30 01:20:23.672685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.672676    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee055e72-2e4b-4e4f-82c4-9649c6dffb11" containerName="revoke-keys"
Oct 30 01:20:23.672722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.672695    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="be997695-27a5-4279-96a0-35cac451f1f8" containerName="revoke-keys"
Oct 30 01:20:23.677817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.677787    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.694290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.694260    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"]
Oct 30 01:20:23.699068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.698995    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:23.699256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.699007    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:23.699658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.699638    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:20:23.702867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.702830    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:23.762173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.762131    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.762354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.762262    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.762444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.762426    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqbd\" (UniqueName: \"kubernetes.io/projected/27cb232f-7fad-42f8-8219-cbf56755b74f-kube-api-access-pdqbd\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.864261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.864219    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqbd\" (UniqueName: \"kubernetes.io/projected/27cb232f-7fad-42f8-8219-cbf56755b74f-kube-api-access-pdqbd\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.864693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.864657    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.864812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.864766    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.865744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.865718    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.867491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.867449    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.938529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.936849    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqbd\" (UniqueName: \"kubernetes.io/projected/27cb232f-7fad-42f8-8219-cbf56755b74f-kube-api-access-pdqbd\") pod \"maas-api-revoke-sub-65f57134-vz7ns\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:23.997030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:23.996993    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:24.311218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:24.311160    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"]
Oct 30 01:20:24.312190 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:24.312160    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cb232f_7fad_42f8_8219_cbf56755b74f.slice/crio-f55b8a73f84109913d8612e3f26553f9f732d38101cb5e0afc672de99103d21c WatchSource:0}: Error finding container f55b8a73f84109913d8612e3f26553f9f732d38101cb5e0afc672de99103d21c: Status 404 returned error can't find the container with id f55b8a73f84109913d8612e3f26553f9f732d38101cb5e0afc672de99103d21c
Oct 30 01:20:25.121333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.121243    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"]
Oct 30 01:20:25.126168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.126145    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.147067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.147024    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"]
Oct 30 01:20:25.185662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.185629    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.186120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.186079    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89bb\" (UniqueName: \"kubernetes.io/projected/9510bd79-7dd9-414a-ae0d-ea6d775a1970-kube-api-access-p89bb\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.186286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.186193    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.288194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.288141    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89bb\" (UniqueName: \"kubernetes.io/projected/9510bd79-7dd9-414a-ae0d-ea6d775a1970-kube-api-access-p89bb\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.288391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.288249    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.288391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.288292    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.289067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.289035    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.290907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.290885    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.302258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.302225    5104 generic.go:356] "Generic (PLEG): container finished" podID="27cb232f-7fad-42f8-8219-cbf56755b74f" containerID="6530257aadc76e6ddc1a608ffd84f9f7be6f7ecadcad2d47be8874074d473743" exitCode=0
Oct 30 01:20:25.302434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.302311    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns" event={"ID":"27cb232f-7fad-42f8-8219-cbf56755b74f","Type":"ContainerDied","Data":"6530257aadc76e6ddc1a608ffd84f9f7be6f7ecadcad2d47be8874074d473743"}
Oct 30 01:20:25.302434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.302378    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns" event={"ID":"27cb232f-7fad-42f8-8219-cbf56755b74f","Type":"ContainerStarted","Data":"f55b8a73f84109913d8612e3f26553f9f732d38101cb5e0afc672de99103d21c"}
Oct 30 01:20:25.307931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.307904    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89bb\" (UniqueName: \"kubernetes.io/projected/9510bd79-7dd9-414a-ae0d-ea6d775a1970-kube-api-access-p89bb\") pod \"maas-api-revoke-sub-eba142b9-pj586\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.448155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.448112    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:25.765692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:25.765654    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"]
Oct 30 01:20:25.766082 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:25.766053    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9510bd79_7dd9_414a_ae0d_ea6d775a1970.slice/crio-5edad1c9c9c91bb664e54344c60ffde1ce267077af6dbfc01188def5e46d1355 WatchSource:0}: Error finding container 5edad1c9c9c91bb664e54344c60ffde1ce267077af6dbfc01188def5e46d1355: Status 404 returned error can't find the container with id 5edad1c9c9c91bb664e54344c60ffde1ce267077af6dbfc01188def5e46d1355
Oct 30 01:20:26.312809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.312768    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586" event={"ID":"9510bd79-7dd9-414a-ae0d-ea6d775a1970","Type":"ContainerStarted","Data":"1ae419b5009d2a85044fe8e62ea227b4a7504b4aecf8920581dc1a1d0d7e4785"}
Oct 30 01:20:26.313306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.312826    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586" event={"ID":"9510bd79-7dd9-414a-ae0d-ea6d775a1970","Type":"ContainerStarted","Data":"5edad1c9c9c91bb664e54344c60ffde1ce267077af6dbfc01188def5e46d1355"}
Oct 30 01:20:26.350332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.350214    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586" podStartSLOduration=1.350191286 podStartE2EDuration="1.350191286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:26.347742588 +0000 UTC m=+2267.895433083" watchObservedRunningTime="2026-10-30 01:20:26.350191286 +0000 UTC m=+2267.897881781"
Oct 30 01:20:26.497947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.497914    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:26.614040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.613949    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-openshift-ca\") pod \"27cb232f-7fad-42f8-8219-cbf56755b74f\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") "
Oct 30 01:20:26.614040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.614025    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27cb232f-7fad-42f8-8219-cbf56755b74f-kube-api-access-pdqbd\" (UniqueName: \"kubernetes.io/projected/27cb232f-7fad-42f8-8219-cbf56755b74f-kube-api-access-pdqbd\") pod \"27cb232f-7fad-42f8-8219-cbf56755b74f\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") "
Oct 30 01:20:26.614350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.614327    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-xks-ca\") pod \"27cb232f-7fad-42f8-8219-cbf56755b74f\" (UID: \"27cb232f-7fad-42f8-8219-cbf56755b74f\") "
Oct 30 01:20:26.614449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.614419    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-openshift-ca" pod "27cb232f-7fad-42f8-8219-cbf56755b74f" (UID: "27cb232f-7fad-42f8-8219-cbf56755b74f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:26.614970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.614944    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:26.616311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.616288    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-xks-ca" pod "27cb232f-7fad-42f8-8219-cbf56755b74f" (UID: "27cb232f-7fad-42f8-8219-cbf56755b74f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:26.616491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.616470    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cb232f-7fad-42f8-8219-cbf56755b74f-kube-api-access-pdqbd" pod "27cb232f-7fad-42f8-8219-cbf56755b74f" (UID: "27cb232f-7fad-42f8-8219-cbf56755b74f"). InnerVolumeSpecName "kube-api-access-pdqbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:26.716582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.716540    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cb232f-7fad-42f8-8219-cbf56755b74f-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:26.716582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:26.716575    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdqbd\" (UniqueName: \"kubernetes.io/projected/27cb232f-7fad-42f8-8219-cbf56755b74f-kube-api-access-pdqbd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.323557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:27.323521    5104 generic.go:356] "Generic (PLEG): container finished" podID="9510bd79-7dd9-414a-ae0d-ea6d775a1970" containerID="1ae419b5009d2a85044fe8e62ea227b4a7504b4aecf8920581dc1a1d0d7e4785" exitCode=0
Oct 30 01:20:27.324035 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:27.323602    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586" event={"ID":"9510bd79-7dd9-414a-ae0d-ea6d775a1970","Type":"ContainerDied","Data":"1ae419b5009d2a85044fe8e62ea227b4a7504b4aecf8920581dc1a1d0d7e4785"}
Oct 30 01:20:27.325155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:27.325129    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns" event={"ID":"27cb232f-7fad-42f8-8219-cbf56755b74f","Type":"ContainerDied","Data":"f55b8a73f84109913d8612e3f26553f9f732d38101cb5e0afc672de99103d21c"}
Oct 30 01:20:27.325271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:27.325162    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"
Oct 30 01:20:27.325271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:27.325178    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55b8a73f84109913d8612e3f26553f9f732d38101cb5e0afc672de99103d21c"
Oct 30 01:20:28.540510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.540478    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:28.646760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.646667    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9510bd79-7dd9-414a-ae0d-ea6d775a1970-kube-api-access-p89bb\" (UniqueName: \"kubernetes.io/projected/9510bd79-7dd9-414a-ae0d-ea6d775a1970-kube-api-access-p89bb\") pod \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") "
Oct 30 01:20:28.646930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.646790    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-xks-ca\") pod \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") "
Oct 30 01:20:28.646930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.646899    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-openshift-ca\") pod \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\" (UID: \"9510bd79-7dd9-414a-ae0d-ea6d775a1970\") "
Oct 30 01:20:28.647291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.647261    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-openshift-ca" pod "9510bd79-7dd9-414a-ae0d-ea6d775a1970" (UID: "9510bd79-7dd9-414a-ae0d-ea6d775a1970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:28.647777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.647751    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.649538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.649509    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-xks-ca" pod "9510bd79-7dd9-414a-ae0d-ea6d775a1970" (UID: "9510bd79-7dd9-414a-ae0d-ea6d775a1970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:28.649638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.649593    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9510bd79-7dd9-414a-ae0d-ea6d775a1970-kube-api-access-p89bb" pod "9510bd79-7dd9-414a-ae0d-ea6d775a1970" (UID: "9510bd79-7dd9-414a-ae0d-ea6d775a1970"). InnerVolumeSpecName "kube-api-access-p89bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:28.748897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.748859    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p89bb\" (UniqueName: \"kubernetes.io/projected/9510bd79-7dd9-414a-ae0d-ea6d775a1970-kube-api-access-p89bb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.748897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:28.748894    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9510bd79-7dd9-414a-ae0d-ea6d775a1970-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:29.345865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:29.345834    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"
Oct 30 01:20:29.346089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:29.345871    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586" event={"ID":"9510bd79-7dd9-414a-ae0d-ea6d775a1970","Type":"ContainerDied","Data":"5edad1c9c9c91bb664e54344c60ffde1ce267077af6dbfc01188def5e46d1355"}
Oct 30 01:20:29.346089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:29.345926    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5edad1c9c9c91bb664e54344c60ffde1ce267077af6dbfc01188def5e46d1355"
Oct 30 01:20:35.941543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.941506    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"]
Oct 30 01:20:35.942333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.942315    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9510bd79-7dd9-414a-ae0d-ea6d775a1970" containerName="revoke-keys"
Oct 30 01:20:35.942387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.942335    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9510bd79-7dd9-414a-ae0d-ea6d775a1970" containerName="revoke-keys"
Oct 30 01:20:35.942387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.942371    5104 cpu_manager.go:414] "containerMap: removing container" podUID="27cb232f-7fad-42f8-8219-cbf56755b74f" containerName="revoke-keys"
Oct 30 01:20:35.942387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.942377    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27cb232f-7fad-42f8-8219-cbf56755b74f" containerName="revoke-keys"
Oct 30 01:20:35.942540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.942530    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="27cb232f-7fad-42f8-8219-cbf56755b74f" containerName="revoke-keys"
Oct 30 01:20:35.942578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.942548    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9510bd79-7dd9-414a-ae0d-ea6d775a1970" containerName="revoke-keys"
Oct 30 01:20:35.946534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.946512    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:35.963835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.963802    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"]
Oct 30 01:20:35.969345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.969308    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:35.969345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.969314    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:35.969622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.969371    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:35.971446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:35.971425    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:20:36.104208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.104161    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nsxj\" (UniqueName: \"kubernetes.io/projected/19921e74-8ff6-44ec-ad17-cdd9207d9848-kube-api-access-8nsxj\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.104473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.104429    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.104618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.104595    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.206385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.206273    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.206600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.206405    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.206600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.206572    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nsxj\" (UniqueName: \"kubernetes.io/projected/19921e74-8ff6-44ec-ad17-cdd9207d9848-kube-api-access-8nsxj\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.207029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.207006    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.208941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.208914    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.224790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.224753    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nsxj\" (UniqueName: \"kubernetes.io/projected/19921e74-8ff6-44ec-ad17-cdd9207d9848-kube-api-access-8nsxj\") pod \"maas-api-revoke-sub-cb7ff1cd-mzzjv\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.261524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.261485    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:36.579425 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:36.579373    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19921e74_8ff6_44ec_ad17_cdd9207d9848.slice/crio-9616080b8940eff3fd15b30d59eb97436c8dd1eb2d8a4f525f662af3f7e60e07 WatchSource:0}: Error finding container 9616080b8940eff3fd15b30d59eb97436c8dd1eb2d8a4f525f662af3f7e60e07: Status 404 returned error can't find the container with id 9616080b8940eff3fd15b30d59eb97436c8dd1eb2d8a4f525f662af3f7e60e07
Oct 30 01:20:36.589385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.589185    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"]
Oct 30 01:20:36.941684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:36.941647    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"]
Oct 30 01:20:37.426129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:37.426095    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv" event={"ID":"19921e74-8ff6-44ec-ad17-cdd9207d9848","Type":"ContainerStarted","Data":"8549f3acd01abc886c4d71f6060508c87096eb2a6a79131206d55719fa499682"}
Oct 30 01:20:37.426334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:37.426145    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv" event={"ID":"19921e74-8ff6-44ec-ad17-cdd9207d9848","Type":"ContainerStarted","Data":"9616080b8940eff3fd15b30d59eb97436c8dd1eb2d8a4f525f662af3f7e60e07"}
Oct 30 01:20:37.476450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:37.476381    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv" podStartSLOduration=2.476358957 podStartE2EDuration="2.476358957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:37.46384013 +0000 UTC m=+2279.011530628" watchObservedRunningTime="2026-10-30 01:20:37.476358957 +0000 UTC m=+2279.024049456"
Oct 30 01:20:37.957007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:37.956919    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"]
Oct 30 01:20:37.969831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:37.969789    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:37.988879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:37.988846    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d32079-openshift-default-dockercfg-c4x75\""
Oct 30 01:20:37.989963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:37.989932    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"]
Oct 30 01:20:38.037547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.037509    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-socket\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.037725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.037569    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jmb\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-kube-api-access-c6jmb\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.037725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.037696    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-istio-token\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.038099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.037739    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-certs\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.038099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.037778    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c888e628-a915-4c08-b192-10976b6148cc-istiod-ca-cert\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.038099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.037892    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-envoy\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.038099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.037923    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c888e628-a915-4c08-b192-10976b6148cc-istio-podinfo\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.038099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.038057    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-credential-socket\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.038435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.038140    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-data\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151158    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-socket\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151230    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jmb\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-kube-api-access-c6jmb\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151282    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-istio-token\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151311    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-certs\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151364    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c888e628-a915-4c08-b192-10976b6148cc-istiod-ca-cert\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151414    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-envoy\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151444    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c888e628-a915-4c08-b192-10976b6148cc-istio-podinfo\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151535    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-credential-socket\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.151590    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-data\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.152110    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-data\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.152359    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-socket\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.158481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.153210    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-certs\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.193527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.169073    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c888e628-a915-4c08-b192-10976b6148cc-istiod-ca-cert\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.193527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.169370    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-credential-socket\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.193527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.193250    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c888e628-a915-4c08-b192-10976b6148cc-istio-podinfo\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.216576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.204943    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-envoy\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.225490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.225168    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-istio-token\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.240494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.233348    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jmb\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-kube-api-access-c6jmb\") pod \"e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") " pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.337498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.332780    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:38.444031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.443990    5104 generic.go:356] "Generic (PLEG): container finished" podID="19921e74-8ff6-44ec-ad17-cdd9207d9848" containerID="8549f3acd01abc886c4d71f6060508c87096eb2a6a79131206d55719fa499682" exitCode=0
Oct 30 01:20:38.444636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.444601    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv" event={"ID":"19921e74-8ff6-44ec-ad17-cdd9207d9848","Type":"ContainerDied","Data":"8549f3acd01abc886c4d71f6060508c87096eb2a6a79131206d55719fa499682"}
Oct 30 01:20:38.809358 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:38.809316    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc888e628_a915_4c08_b192_10976b6148cc.slice/crio-9556c90a154c798882a7beeee0ed1b447a4dd0f95940dda157914fd2b55564ea WatchSource:0}: Error finding container 9556c90a154c798882a7beeee0ed1b447a4dd0f95940dda157914fd2b55564ea: Status 404 returned error can't find the container with id 9556c90a154c798882a7beeee0ed1b447a4dd0f95940dda157914fd2b55564ea
Oct 30 01:20:38.811370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.811339    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"]
Oct 30 01:20:38.813546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.813504    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:38.813674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.813592    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:38.813674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:38.813635    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:39.111568 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:20:39.109904    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:20:39.454707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.454671    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" event={"ID":"c888e628-a915-4c08-b192-10976b6148cc","Type":"ContainerStarted","Data":"f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5"}
Oct 30 01:20:39.454904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.454733    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" event={"ID":"c888e628-a915-4c08-b192-10976b6148cc","Type":"ContainerStarted","Data":"9556c90a154c798882a7beeee0ed1b447a4dd0f95940dda157914fd2b55564ea"}
Oct 30 01:20:39.520304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.520235    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podStartSLOduration=2.520214887 podStartE2EDuration="2.520214887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:39.504687202 +0000 UTC m=+2281.052377700" watchObservedRunningTime="2026-10-30 01:20:39.520214887 +0000 UTC m=+2281.067905385"
Oct 30 01:20:39.737693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.737664    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:39.809128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.809092    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19921e74-8ff6-44ec-ad17-cdd9207d9848-kube-api-access-8nsxj\" (UniqueName: \"kubernetes.io/projected/19921e74-8ff6-44ec-ad17-cdd9207d9848-kube-api-access-8nsxj\") pod \"19921e74-8ff6-44ec-ad17-cdd9207d9848\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") "
Oct 30 01:20:39.809350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.809253    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-openshift-ca\") pod \"19921e74-8ff6-44ec-ad17-cdd9207d9848\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") "
Oct 30 01:20:39.809350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.809330    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-xks-ca\") pod \"19921e74-8ff6-44ec-ad17-cdd9207d9848\" (UID: \"19921e74-8ff6-44ec-ad17-cdd9207d9848\") "
Oct 30 01:20:39.810168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.810132    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-openshift-ca" pod "19921e74-8ff6-44ec-ad17-cdd9207d9848" (UID: "19921e74-8ff6-44ec-ad17-cdd9207d9848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:39.813605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.813576    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-xks-ca" pod "19921e74-8ff6-44ec-ad17-cdd9207d9848" (UID: "19921e74-8ff6-44ec-ad17-cdd9207d9848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:39.820508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.820347    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19921e74-8ff6-44ec-ad17-cdd9207d9848-kube-api-access-8nsxj" pod "19921e74-8ff6-44ec-ad17-cdd9207d9848" (UID: "19921e74-8ff6-44ec-ad17-cdd9207d9848"). InnerVolumeSpecName "kube-api-access-8nsxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:39.932915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.932871    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.933277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.933252    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19921e74-8ff6-44ec-ad17-cdd9207d9848-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.933532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:39.933505    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nsxj\" (UniqueName: \"kubernetes.io/projected/19921e74-8ff6-44ec-ad17-cdd9207d9848-kube-api-access-8nsxj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:40.333945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:40.333903    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:40.340665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:40.340636    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:40.465796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:40.465761    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"
Oct 30 01:20:40.465796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:40.465786    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv" event={"ID":"19921e74-8ff6-44ec-ad17-cdd9207d9848","Type":"ContainerDied","Data":"9616080b8940eff3fd15b30d59eb97436c8dd1eb2d8a4f525f662af3f7e60e07"}
Oct 30 01:20:40.466068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:40.465845    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9616080b8940eff3fd15b30d59eb97436c8dd1eb2d8a4f525f662af3f7e60e07"
Oct 30 01:20:40.466273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:40.466255    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:40.467430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:40.467412    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:20:41.575480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.569904    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"]
Oct 30 01:20:41.575480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.571102    5104 cpu_manager.go:414] "containerMap: removing container" podUID="19921e74-8ff6-44ec-ad17-cdd9207d9848" containerName="revoke-keys"
Oct 30 01:20:41.575480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.571126    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19921e74-8ff6-44ec-ad17-cdd9207d9848" containerName="revoke-keys"
Oct 30 01:20:41.575480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.571422    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="19921e74-8ff6-44ec-ad17-cdd9207d9848" containerName="revoke-keys"
Oct 30 01:20:41.597983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.597425    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.599493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.598424    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"]
Oct 30 01:20:41.613083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.613060    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-262230-openshift-default-dockercfg-ks9dj\""
Oct 30 01:20:41.659765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.659728    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.659953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.659883    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.659953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.659945    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.660109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.660089    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.660197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.660176    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.660310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.660292    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdg5\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-kube-api-access-2pdg5\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.660406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.660389    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.660475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.660429    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.660551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.660486    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.761925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.761885    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.761995    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762024    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762064    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762116    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdg5\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-kube-api-access-2pdg5\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762177    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762203    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762243    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762389    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762559    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762598    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762683    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762868    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.762976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.762920    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.764506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.764478    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.765227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.765206    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.781297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.781265    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdg5\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-kube-api-access-2pdg5\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.781501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.781323    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:41.915752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:41.915641    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:42.241628 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:42.241584    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39f2e47_e349_4f0d_b7ec_9ef1891888ec.slice/crio-ada1188a60577b476d94c13b0c271fff9040065a25baf4bd3984c86ae770a453 WatchSource:0}: Error finding container ada1188a60577b476d94c13b0c271fff9040065a25baf4bd3984c86ae770a453: Status 404 returned error can't find the container with id ada1188a60577b476d94c13b0c271fff9040065a25baf4bd3984c86ae770a453
Oct 30 01:20:42.241795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.241631    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"]
Oct 30 01:20:42.244924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.244876    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:42.245082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.244984    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:42.245082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.245030    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:20:42.492605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.492500    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" event={"ID":"d39f2e47-e349-4f0d-b7ec-9ef1891888ec","Type":"ContainerStarted","Data":"543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1"}
Oct 30 01:20:42.492605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.492573    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" event={"ID":"d39f2e47-e349-4f0d-b7ec-9ef1891888ec","Type":"ContainerStarted","Data":"ada1188a60577b476d94c13b0c271fff9040065a25baf4bd3984c86ae770a453"}
Oct 30 01:20:42.534953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.534884    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podStartSLOduration=1.534862046 podStartE2EDuration="1.534862046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:42.532715998 +0000 UTC m=+2284.080406494" watchObservedRunningTime="2026-10-30 01:20:42.534862046 +0000 UTC m=+2284.082552542"
Oct 30 01:20:42.916100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:42.916005    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:43.709481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:43.707386    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"]
Oct 30 01:20:43.714884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:43.714853    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:43.717269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:43.717223    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"]
Oct 30 01:20:43.729614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:43.729552    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d32079\""
Oct 30 01:20:43.891561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:43.887576    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxshc\" (UniqueName: \"kubernetes.io/projected/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-kube-api-access-lxshc\") pod \"payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:43.891561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:43.887950    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:44.000534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.000274    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d32079-dockercfg-pp7wt\""
Oct 30 01:20:44.042477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.033223    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxshc\" (UniqueName: \"kubernetes.io/projected/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-kube-api-access-lxshc\") pod \"payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:44.042477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.033607    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:44.120692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.120640    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:44.121019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.120976    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:44.123600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.123535    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:44.146140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.146103    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"]
Oct 30 01:20:44.167679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.167645    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"]
Oct 30 01:20:44.168075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.168010    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.169663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.168357    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxshc\" (UniqueName: \"kubernetes.io/projected/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-kube-api-access-lxshc\") pod \"payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:44.255524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.255428    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7576h\" (UniqueName: \"kubernetes.io/projected/00b50f17-564f-409d-a863-2052bb8a6de3-kube-api-access-7576h\") pod \"payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.255524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.255520    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b50f17-564f-409d-a863-2052bb8a6de3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.337991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.337946    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:44.357142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.357105    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7576h\" (UniqueName: \"kubernetes.io/projected/00b50f17-564f-409d-a863-2052bb8a6de3-kube-api-access-7576h\") pod \"payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.357288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.357218    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b50f17-564f-409d-a863-2052bb8a6de3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.358329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.358296    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b50f17-564f-409d-a863-2052bb8a6de3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.389312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.389212    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7576h\" (UniqueName: \"kubernetes.io/projected/00b50f17-564f-409d-a863-2052bb8a6de3-kube-api-access-7576h\") pod \"payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.537702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.537612    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:44.763704 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:44.762584    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5c39b0_ada0_4740_b2db_0f17a8f557f8.slice/crio-7edcbe212efcac73a07a008ab6036f067a453bc7ec9693b10f98850301d15214 WatchSource:0}: Error finding container 7edcbe212efcac73a07a008ab6036f067a453bc7ec9693b10f98850301d15214: Status 404 returned error can't find the container with id 7edcbe212efcac73a07a008ab6036f067a453bc7ec9693b10f98850301d15214
Oct 30 01:20:44.931521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.855878    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"]
Oct 30 01:20:44.965497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.962004    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:44.965497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:44.962088    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:45.112162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.112120    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"]
Oct 30 01:20:45.123820 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:20:45.121921    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b50f17_564f_409d_a863_2052bb8a6de3.slice/crio-709a4a991df87c9c669eeb899c77f0d4250df23392a8ef788252d8f2ffc94b4d WatchSource:0}: Error finding container 709a4a991df87c9c669eeb899c77f0d4250df23392a8ef788252d8f2ffc94b4d: Status 404 returned error can't find the container with id 709a4a991df87c9c669eeb899c77f0d4250df23392a8ef788252d8f2ffc94b4d
Oct 30 01:20:45.680482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.670110    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc" event={"ID":"00b50f17-564f-409d-a863-2052bb8a6de3","Type":"ContainerStarted","Data":"80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16"}
Oct 30 01:20:45.680482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.670189    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc" event={"ID":"00b50f17-564f-409d-a863-2052bb8a6de3","Type":"ContainerStarted","Data":"709a4a991df87c9c669eeb899c77f0d4250df23392a8ef788252d8f2ffc94b4d"}
Oct 30 01:20:45.680482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.670251    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:45.707014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.706877    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm" event={"ID":"8a5c39b0-ada0-4740-b2db-0f17a8f557f8","Type":"ContainerStarted","Data":"d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12"}
Oct 30 01:20:45.707014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.706947    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm" event={"ID":"8a5c39b0-ada0-4740-b2db-0f17a8f557f8","Type":"ContainerStarted","Data":"7edcbe212efcac73a07a008ab6036f067a453bc7ec9693b10f98850301d15214"}
Oct 30 01:20:45.715481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.707937    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:45.756507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.755001    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc" podStartSLOduration=2.754979896 podStartE2EDuration="2.754979896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:45.749862374 +0000 UTC m=+2287.297552870" watchObservedRunningTime="2026-10-30 01:20:45.754979896 +0000 UTC m=+2287.302670396"
Oct 30 01:20:45.804590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.804513    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm" podStartSLOduration=2.804493483 podStartE2EDuration="2.804493483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:45.794930445 +0000 UTC m=+2287.342620970" watchObservedRunningTime="2026-10-30 01:20:45.804493483 +0000 UTC m=+2287.352183977"
Oct 30 01:20:45.919483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.918180    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:45.919483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:45.918285    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:46.529489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.528633    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"]
Oct 30 01:20:46.601481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.596590    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:46.658609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.635914    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:46.658609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.636066    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:46.658609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.636134    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:20:46.658609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.636167    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:46.759399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.750181    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:46.759399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.750332    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wh2\" (UniqueName: \"kubernetes.io/projected/9750ee0b-7611-40e2-81af-355a67d3634b-kube-api-access-86wh2\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:46.759399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.750602    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:46.953539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.951771    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:46.953539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.951929    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:46.953539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.952025    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wh2\" (UniqueName: \"kubernetes.io/projected/9750ee0b-7611-40e2-81af-355a67d3634b-kube-api-access-86wh2\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:47.143485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.987577    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"]
Oct 30 01:20:47.143485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:46.990653    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:47.165488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.147992    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:47.220542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.195573    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:47.257480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.233544    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:47.257480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.233615    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:47.418638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.397118    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:20:47.514176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.455711    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:47.551838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.551786    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:47.597413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.595999    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wh2\" (UniqueName: \"kubernetes.io/projected/9750ee0b-7611-40e2-81af-355a67d3634b-kube-api-access-86wh2\") pod \"maas-api-revoke-sub-82cf0615-rsg6c\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:47.597413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.596661    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:47.597413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.596948    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:20:47.833637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:47.825026    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:48.103542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.085898    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.133483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.132589    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:48.133483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.132670    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:48.202495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.194557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.207835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.204166    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:48.211726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.211438    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:48.212933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.212881    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:20:48.248479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.248414    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.275853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.275805    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Oct 30 01:20:48.725494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.716968    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:48.725494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.425865    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.725494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.602349    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.725494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.602414    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.725494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.602437    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.725494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.602980    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:20:48.763948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.347214    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:48.763948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.612924    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:48.824625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.824570    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:48.976582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.935792    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:48.976582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:48.935871    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:49.169473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.129503    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.657062    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.695209    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.695444    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.695625    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": context deadline exceeded"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.720581    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.720812    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.720890    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.720974    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:20:49.810916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.746092    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.005481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:49.983706    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.024230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.010569    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb" podUID="626e0a50-e9f7-4b0f-aa09-226cba9a0dfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.348422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.272408    5104 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-wfw4k container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:20:50.348422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.342892    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k" podUID="7a355cec-6892-4b36-89b7-60c6b3460bf0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.502486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.482656    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:20:50.502486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.482718    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.557909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.557860    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:50.558243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.558189    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:50.563488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.562865    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.600555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.600500    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.661957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.661905    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.749616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.749569    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.779970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.779270    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:20:50.779970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.779671    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:20:50.780113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.779965    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:20:50.826976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.826911    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:20:50.831488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.830528    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:50.831488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.831045    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": context deadline exceeded"
Oct 30 01:20:50.840502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.835555    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:20:50.840502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.835680    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:20:50.890489 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:20:50.876969    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.186s"
Oct 30 01:20:50.969548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.963653    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"]
Oct 30 01:20:50.998495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.989572    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:20:50.998495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.989657    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:20:51.000923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:50.916967    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 01:20:51.001133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.001110    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 01:20:51.349582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.349293    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:20:51.349582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.349369    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:20:51.349582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.349410    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:20:51.349582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.349430    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:20:51.377838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.369197    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:51.627128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.627044    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:20:51.692759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.692711    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:20:51.701487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.693162    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:20:51.805547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.790895    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:20:51.805547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.790944    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:20:51.805547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.790994    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:20:51.805547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.791034    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:20:51.805547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.791073    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:20:51.805547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.694054    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:51.953689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.922387    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:20:51.953689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.937401    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:20:51.957571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.957521    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:20:51.994658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.994597    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:20:52.064725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.064668    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:20:52.077955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.957665    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": context deadline exceeded"
Oct 30 01:20:52.077955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.067078    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:20:52.077955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.958561    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:52.077955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.067343    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:20:52.077955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.957611    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:20:52.247540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.244078    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" containerID="cri-o://f7a805aee37fc64bcb1cf8d0f93c925daf42bb40e7c4722766cae4a2e1325b01" gracePeriod=45
Oct 30 01:20:52.247848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:51.957644    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:20:52.300937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.300895    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:20:52.321489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.310133    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c" event={"ID":"9750ee0b-7611-40e2-81af-355a67d3634b","Type":"ContainerStarted","Data":"73c2c523af9c292aa31388447091a059aaf24dda8dae4ff900419c0da67a9ff6"}
Oct 30 01:20:52.567879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.567132    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:20:52.567879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.331298    5104 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 01:20:52.568167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.396195    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:20:52.665490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.664610    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:52.767713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.767627    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": read tcp 10.132.0.2:37100->10.132.1.28:8000: read: connection reset by peer"
Oct 30 01:20:52.833569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.810543    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:20:52.933811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.933759    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:52.973674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:52.973623    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:20:53.003590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.003546    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:20:53.022500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.011622    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 01:20:53.022500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.011871    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:20:53.022500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.012177    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:20:53.022500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.014498    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": dial tcp 10.132.1.28:8000: connect: connection refused"
Oct 30 01:20:53.095842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.095733    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:53.112480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.106845    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:20:53.189525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.188675    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:53.245753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.219634    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:53.388572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.375296    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:20:53.428094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.424767    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:53.488496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.468190    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:53.507375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.507348    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:20:53.534497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.531519    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:53.570347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.570311    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:20:53.571294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.571246    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:53.605477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.599163    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:20:53.605756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.605727    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:20:53.897637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.897573    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:53.960489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.957421    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:20:53.989479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.989016    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:20:54.036482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:53.992382    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": dial tcp 10.132.1.28:8000: connect: connection refused"
Oct 30 01:20:54.036482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.001276    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:20:54.036482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.009022    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:20:54.036482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.009060    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:54.036482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.022575    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:20:54.155813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.134571    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:54.239257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.229872    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:54.239257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.229738    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:54.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.312578    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:20:54.400553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.375733    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:20:54.400553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.376184    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:20:54.506724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.503263    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:20:54.539711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.538623    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:20:54.539711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.539320    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:20:54.542848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.542815    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:20:54.768614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.747109    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:20:54.768935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.768913    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:20:54.769124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.769112    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:20:54.782937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.782901    5104 generic.go:356] "Generic (PLEG): container finished" podID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerID="f7a805aee37fc64bcb1cf8d0f93c925daf42bb40e7c4722766cae4a2e1325b01" exitCode=0
Oct 30 01:20:54.789966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.789925    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" event={"ID":"731562c2-d399-49ee-9e12-cecfb5133e6f","Type":"ContainerDied","Data":"f7a805aee37fc64bcb1cf8d0f93c925daf42bb40e7c4722766cae4a2e1325b01"}
Oct 30 01:20:54.850023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.847485    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c" event={"ID":"9750ee0b-7611-40e2-81af-355a67d3634b","Type":"ContainerStarted","Data":"bf549b5eba9aa074c222c2d015687b60bb7f96c00f6567d732c158f44420cf34"}
Oct 30 01:20:54.947598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:54.947543    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": dial tcp 10.132.1.28:8000: connect: connection refused"
Oct 30 01:20:55.203257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.203193    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c" podStartSLOduration=9.203169214 podStartE2EDuration="9.203169214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:55.009842437 +0000 UTC m=+2296.557532929" watchObservedRunningTime="2026-10-30 01:20:55.203169214 +0000 UTC m=+2296.750859711"
Oct 30 01:20:55.277082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.277042    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:20:55.435023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.434971    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-dshm\") pod \"731562c2-d399-49ee-9e12-cecfb5133e6f\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") "
Oct 30 01:20:55.435241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.435094    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731562c2-d399-49ee-9e12-cecfb5133e6f-kube-api-access-hj6wp\" (UniqueName: \"kubernetes.io/projected/731562c2-d399-49ee-9e12-cecfb5133e6f-kube-api-access-hj6wp\") pod \"731562c2-d399-49ee-9e12-cecfb5133e6f\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") "
Oct 30 01:20:55.435241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.435160    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-tmp-dir\") pod \"731562c2-d399-49ee-9e12-cecfb5133e6f\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") "
Oct 30 01:20:55.435241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.435215    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731562c2-d399-49ee-9e12-cecfb5133e6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/731562c2-d399-49ee-9e12-cecfb5133e6f-tls-certs\") pod \"731562c2-d399-49ee-9e12-cecfb5133e6f\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") "
Oct 30 01:20:55.435432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.435341    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-home\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-home\") pod \"731562c2-d399-49ee-9e12-cecfb5133e6f\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") "
Oct 30 01:20:55.435432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.435409    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-model-cache\") pod \"731562c2-d399-49ee-9e12-cecfb5133e6f\" (UID: \"731562c2-d399-49ee-9e12-cecfb5133e6f\") "
Oct 30 01:20:55.436134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.436100    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-tmp-dir" pod "731562c2-d399-49ee-9e12-cecfb5133e6f" (UID: "731562c2-d399-49ee-9e12-cecfb5133e6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:55.436236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.436198    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-home" pod "731562c2-d399-49ee-9e12-cecfb5133e6f" (UID: "731562c2-d399-49ee-9e12-cecfb5133e6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:55.438210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.437703    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-model-cache" pod "731562c2-d399-49ee-9e12-cecfb5133e6f" (UID: "731562c2-d399-49ee-9e12-cecfb5133e6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:55.438954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.438910    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-dshm" pod "731562c2-d399-49ee-9e12-cecfb5133e6f" (UID: "731562c2-d399-49ee-9e12-cecfb5133e6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:55.439482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.439422    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731562c2-d399-49ee-9e12-cecfb5133e6f-tls-certs" pod "731562c2-d399-49ee-9e12-cecfb5133e6f" (UID: "731562c2-d399-49ee-9e12-cecfb5133e6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:55.445018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.442209    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731562c2-d399-49ee-9e12-cecfb5133e6f-kube-api-access-hj6wp" pod "731562c2-d399-49ee-9e12-cecfb5133e6f" (UID: "731562c2-d399-49ee-9e12-cecfb5133e6f"). InnerVolumeSpecName "kube-api-access-hj6wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:55.537938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.537896    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731562c2-d399-49ee-9e12-cecfb5133e6f-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:55.537938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.537929    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:55.537938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.537939    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:55.537938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.537947    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:55.538292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.537956    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj6wp\" (UniqueName: \"kubernetes.io/projected/731562c2-d399-49ee-9e12-cecfb5133e6f-kube-api-access-hj6wp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:55.538292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.537967    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731562c2-d399-49ee-9e12-cecfb5133e6f-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:55.859667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.859562    5104 generic.go:356] "Generic (PLEG): container finished" podID="9750ee0b-7611-40e2-81af-355a67d3634b" containerID="bf549b5eba9aa074c222c2d015687b60bb7f96c00f6567d732c158f44420cf34" exitCode=0
Oct 30 01:20:55.859667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.859648    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c" event={"ID":"9750ee0b-7611-40e2-81af-355a67d3634b","Type":"ContainerDied","Data":"bf549b5eba9aa074c222c2d015687b60bb7f96c00f6567d732c158f44420cf34"}
Oct 30 01:20:55.862936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.862901    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56" event={"ID":"731562c2-d399-49ee-9e12-cecfb5133e6f","Type":"ContainerDied","Data":"daa9c5d3ce1aa623065719377a0e658718add47ac447599295e7dbda3d25c306"}
Oct 30 01:20:55.862936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.862954    5104 scope.go:122] "RemoveContainer" containerID="f7a805aee37fc64bcb1cf8d0f93c925daf42bb40e7c4722766cae4a2e1325b01"
Oct 30 01:20:55.863390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.863366    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"
Oct 30 01:20:55.936493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.935641    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"]
Oct 30 01:20:55.944147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:55.944109    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-69dc90/rate-test-model-69dc90-kserve-548b8bb45-prf56"]
Oct 30 01:20:56.588276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:56.587582    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" path="/var/lib/kubelet/pods/731562c2-d399-49ee-9e12-cecfb5133e6f/volumes"
Oct 30 01:20:57.480725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.479267    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:20:57.528010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.527963    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:20:57.575748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.575691    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-openshift-ca\") pod \"9750ee0b-7611-40e2-81af-355a67d3634b\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") "
Oct 30 01:20:57.575748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.575755    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9750ee0b-7611-40e2-81af-355a67d3634b-kube-api-access-86wh2\" (UniqueName: \"kubernetes.io/projected/9750ee0b-7611-40e2-81af-355a67d3634b-kube-api-access-86wh2\") pod \"9750ee0b-7611-40e2-81af-355a67d3634b\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") "
Oct 30 01:20:57.576010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.575850    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-xks-ca\") pod \"9750ee0b-7611-40e2-81af-355a67d3634b\" (UID: \"9750ee0b-7611-40e2-81af-355a67d3634b\") "
Oct 30 01:20:57.587060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.587012    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-openshift-ca" pod "9750ee0b-7611-40e2-81af-355a67d3634b" (UID: "9750ee0b-7611-40e2-81af-355a67d3634b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:57.640172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.610333    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-xks-ca" pod "9750ee0b-7611-40e2-81af-355a67d3634b" (UID: "9750ee0b-7611-40e2-81af-355a67d3634b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:57.640172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.610528    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9750ee0b-7611-40e2-81af-355a67d3634b-kube-api-access-86wh2" pod "9750ee0b-7611-40e2-81af-355a67d3634b" (UID: "9750ee0b-7611-40e2-81af-355a67d3634b"). InnerVolumeSpecName "kube-api-access-86wh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:57.678971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.678924    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:57.678971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.678968    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86wh2\" (UniqueName: \"kubernetes.io/projected/9750ee0b-7611-40e2-81af-355a67d3634b-kube-api-access-86wh2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:57.679180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.678986    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9750ee0b-7611-40e2-81af-355a67d3634b-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:20:57.937691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.937647    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c" event={"ID":"9750ee0b-7611-40e2-81af-355a67d3634b","Type":"ContainerDied","Data":"73c2c523af9c292aa31388447091a059aaf24dda8dae4ff900419c0da67a9ff6"}
Oct 30 01:20:57.937928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.937716    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c2c523af9c292aa31388447091a059aaf24dda8dae4ff900419c0da67a9ff6"
Oct 30 01:20:57.937928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:20:57.937793    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"
Oct 30 01:21:01.787483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.787376    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"]
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.788282    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9750ee0b-7611-40e2-81af-355a67d3634b" containerName="revoke-keys"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.788300    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9750ee0b-7611-40e2-81af-355a67d3634b" containerName="revoke-keys"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.788335    5104 cpu_manager.go:414] "containerMap: removing container" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.788340    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.788494    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="731562c2-d399-49ee-9e12-cecfb5133e6f" containerName="main"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.788524    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9750ee0b-7611-40e2-81af-355a67d3634b" containerName="revoke-keys"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.807886    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.815661    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"]
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.822539    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-999ff9\""
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.860221    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"]
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.878408    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.882303    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndd5\" (UniqueName: \"kubernetes.io/projected/5bde04ee-fb54-4f94-9fa1-23381376ad6c-kube-api-access-4ndd5\") pod \"payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.882907    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde04ee-fb54-4f94-9fa1-23381376ad6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.884860    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"]
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.893779    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-999ff9-dockercfg-bn2sw\""
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.984631    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2xp\" (UniqueName: \"kubernetes.io/projected/e437e532-1c76-4575-885a-01d40c1b4570-kube-api-access-dt2xp\") pod \"payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.984764    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e437e532-1c76-4575-885a-01d40c1b4570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.984817    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndd5\" (UniqueName: \"kubernetes.io/projected/5bde04ee-fb54-4f94-9fa1-23381376ad6c-kube-api-access-4ndd5\") pod \"payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.984965    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde04ee-fb54-4f94-9fa1-23381376ad6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:01.986093    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde04ee-fb54-4f94-9fa1-23381376ad6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.136578    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e437e532-1c76-4575-885a-01d40c1b4570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.136975    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2xp\" (UniqueName: \"kubernetes.io/projected/e437e532-1c76-4575-885a-01d40c1b4570-kube-api-access-dt2xp\") pod \"payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.147541    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e437e532-1c76-4575-885a-01d40c1b4570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.206782    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndd5\" (UniqueName: \"kubernetes.io/projected/5bde04ee-fb54-4f94-9fa1-23381376ad6c-kube-api-access-4ndd5\") pod \"payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.213123    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2xp\" (UniqueName: \"kubernetes.io/projected/e437e532-1c76-4575-885a-01d40c1b4570-kube-api-access-dt2xp\") pod \"payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.223941    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.242447    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"]
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.285552    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"]
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.285653    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.448707    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.448815    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095dd611-ece6-4654-9154-7ae46b0560f2-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.448823    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.448868    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2h5f\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-kube-api-access-d2h5f\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.448935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.449084    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.449194    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.449538    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.449663    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095dd611-ece6-4654-9154-7ae46b0560f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.449792    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551059    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551140    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551194    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095dd611-ece6-4654-9154-7ae46b0560f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551212    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551244    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551295    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095dd611-ece6-4654-9154-7ae46b0560f2-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551324    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2h5f\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-kube-api-access-d2h5f\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551365    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551486    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.551896    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.552348    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.552915    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095dd611-ece6-4654-9154-7ae46b0560f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.555007    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.556695    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.558547    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095dd611-ece6-4654-9154-7ae46b0560f2-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.563409    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.583074    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2h5f\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-kube-api-access-d2h5f\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.606375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.590584    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.618194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.617988    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"]
Oct 30 01:21:02.619633 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:02.619599    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode437e532_1c76_4575_885a_01d40c1b4570.slice/crio-60be3dd85634d662fbd1d397b036d1cd95c1e5650f683d351af2a1f2d4e11a15 WatchSource:0}: Error finding container 60be3dd85634d662fbd1d397b036d1cd95c1e5650f683d351af2a1f2d4e11a15: Status 404 returned error can't find the container with id 60be3dd85634d662fbd1d397b036d1cd95c1e5650f683d351af2a1f2d4e11a15
Oct 30 01:21:02.680815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.680358    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:02.809252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:02.809200    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"]
Oct 30 01:21:02.811593 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:02.811444    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde04ee_fb54_4f94_9fa1_23381376ad6c.slice/crio-bfa09bac01673398cbfe6bc5304a546ab34f882afcbf36e5330f69b1b4d56dfc WatchSource:0}: Error finding container bfa09bac01673398cbfe6bc5304a546ab34f882afcbf36e5330f69b1b4d56dfc: Status 404 returned error can't find the container with id bfa09bac01673398cbfe6bc5304a546ab34f882afcbf36e5330f69b1b4d56dfc
Oct 30 01:21:03.011379 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:03.011342    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095dd611_ece6_4654_9154_7ae46b0560f2.slice/crio-d774c4c0eeeb6d0cf39112a1e6677337008cdb8b5c7505406fb41bf64f895195 WatchSource:0}: Error finding container d774c4c0eeeb6d0cf39112a1e6677337008cdb8b5c7505406fb41bf64f895195: Status 404 returned error can't find the container with id d774c4c0eeeb6d0cf39112a1e6677337008cdb8b5c7505406fb41bf64f895195
Oct 30 01:21:03.012495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.012469    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"]
Oct 30 01:21:03.014888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.014850    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.014987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.014940    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.015068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.014987    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.162239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.162146    5104 generic.go:356] "Generic (PLEG): container finished" podID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerID="6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9" exitCode=0
Oct 30 01:21:03.162239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.162211    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" event={"ID":"4f3a73a2-922c-4ecd-8e7d-4cde4f841835","Type":"ContainerDied","Data":"6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9"}
Oct 30 01:21:03.162751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.162732    5104 scope.go:122] "RemoveContainer" containerID="6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9"
Oct 30 01:21:03.164514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.164451    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn" event={"ID":"5bde04ee-fb54-4f94-9fa1-23381376ad6c","Type":"ContainerStarted","Data":"1557eb8ae86a5ce7c7f6d0d066de9702d6a5c328ecc0fbeb35ca3414c16c5c78"}
Oct 30 01:21:03.164673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.164537    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn" event={"ID":"5bde04ee-fb54-4f94-9fa1-23381376ad6c","Type":"ContainerStarted","Data":"bfa09bac01673398cbfe6bc5304a546ab34f882afcbf36e5330f69b1b4d56dfc"}
Oct 30 01:21:03.164673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.164647    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:03.165104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.165074    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.165210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.165154    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.165210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.165203    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.166603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.166580    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj" event={"ID":"e437e532-1c76-4575-885a-01d40c1b4570","Type":"ContainerStarted","Data":"625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555"}
Oct 30 01:21:03.166722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.166623    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj" event={"ID":"e437e532-1c76-4575-885a-01d40c1b4570","Type":"ContainerStarted","Data":"60be3dd85634d662fbd1d397b036d1cd95c1e5650f683d351af2a1f2d4e11a15"}
Oct 30 01:21:03.166722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.166668    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:03.168927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.168899    5104 generic.go:356] "Generic (PLEG): container finished" podID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerID="3853c548357e366cb193ff58e5e4a1074d0dadedf64d2db788c80115f146ee93" exitCode=0
Oct 30 01:21:03.169057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.169002    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" event={"ID":"6f4f6143-8983-4709-bcc6-01c5def319d2","Type":"ContainerDied","Data":"3853c548357e366cb193ff58e5e4a1074d0dadedf64d2db788c80115f146ee93"}
Oct 30 01:21:03.169434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.169414    5104 scope.go:122] "RemoveContainer" containerID="3853c548357e366cb193ff58e5e4a1074d0dadedf64d2db788c80115f146ee93"
Oct 30 01:21:03.171644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.171622    5104 generic.go:356] "Generic (PLEG): container finished" podID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerID="058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b" exitCode=0
Oct 30 01:21:03.171851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.171663    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" event={"ID":"530a2744-a0ae-46dd-a4c7-ada0d33538d4","Type":"ContainerDied","Data":"058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b"}
Oct 30 01:21:03.172598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.172575    5104 scope.go:122] "RemoveContainer" containerID="058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b"
Oct 30 01:21:03.176254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.176213    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" event={"ID":"095dd611-ece6-4654-9154-7ae46b0560f2","Type":"ContainerStarted","Data":"8de4323acdc17376037bbf9b4767c3265f9069a99134005285d962ee99aab437"}
Oct 30 01:21:03.176374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.176279    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" event={"ID":"095dd611-ece6-4654-9154-7ae46b0560f2","Type":"ContainerStarted","Data":"d774c4c0eeeb6d0cf39112a1e6677337008cdb8b5c7505406fb41bf64f895195"}
Oct 30 01:21:03.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.195288    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.197017    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.197105    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.200484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.197150    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.233584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.233543    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.233887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.233864    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:03.418153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.395119    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:21:03.479851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.479789    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podStartSLOduration=1.479766945 podStartE2EDuration="1.479766945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:03.285300204 +0000 UTC m=+2304.832990702" watchObservedRunningTime="2026-10-30 01:21:03.479766945 +0000 UTC m=+2305.027457442"
Oct 30 01:21:03.505175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.500113    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn" podStartSLOduration=2.500088547 podStartE2EDuration="2.500088547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:03.456940845 +0000 UTC m=+2305.004631343" watchObservedRunningTime="2026-10-30 01:21:03.500088547 +0000 UTC m=+2305.047779044"
Oct 30 01:21:03.602125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.601241    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj" podStartSLOduration=2.601219756 podStartE2EDuration="2.601219756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:03.588087325 +0000 UTC m=+2305.135777837" watchObservedRunningTime="2026-10-30 01:21:03.601219756 +0000 UTC m=+2305.148910253"
Oct 30 01:21:03.698442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:03.693576    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:04.226720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.226565    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" event={"ID":"6f4f6143-8983-4709-bcc6-01c5def319d2","Type":"ContainerStarted","Data":"3e656146389bc57a604490506a0f474ddf3f560281ea067ecc593e52785cefdd"}
Oct 30 01:21:04.228803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.228779    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:21:04.230427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.230391    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:04.230672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.230619    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:04.233555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.233520    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" event={"ID":"530a2744-a0ae-46dd-a4c7-ada0d33538d4","Type":"ContainerStarted","Data":"05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631"}
Oct 30 01:21:04.237502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.237469    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" event={"ID":"4f3a73a2-922c-4ecd-8e7d-4cde4f841835","Type":"ContainerStarted","Data":"85eee898b69c7e27059218d6eaba0cd16bf93aa2c0a3b885c5399fa317d173af"}
Oct 30 01:21:04.363557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.363525    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:21:04.500570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.496936    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:21:04.500570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.497158    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:04.500570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.497208    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:04.687481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.687288    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:04.687481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.687377    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:04.914798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.914646    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:21:04.916938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.916907    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:04.917177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:04.917154    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:05.239479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.238586    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:05.239479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.238656    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:05.507579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.494094    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"]
Oct 30 01:21:05.533532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.522546    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:05.533532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.522615    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:05.611725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.607867    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:05.644848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.644797    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"]
Oct 30 01:21:05.646745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.645212    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:05.707165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.707131    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e92648-openshift-default-dockercfg-zt4vb\""
Oct 30 01:21:05.708052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.708010    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:05.708301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.708267    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:05.885534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.863572    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:21:05.972677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:05.972614    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.063948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.063901    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:06.071752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.071707    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:06.089201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.068873    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.205552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.205506    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbpxb\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-kube-api-access-nbpxb\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.340282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.340238    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.340990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.340952    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.341322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.341300    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.341498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.341477    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.341688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.341671    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.341840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.341822    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.341981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.341960    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.352483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.342322    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b35fbd-906b-450d-99a0-b5b0f405b607-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.364710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.364652    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:21:06.443484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.411569    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.443484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.412271    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.480384    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b35fbd-906b-450d-99a0-b5b0f405b607-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.472907    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.480543    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpxb\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-kube-api-access-nbpxb\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.480968    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.481043    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.481223    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.481260    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.481339    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.481404    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.505637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.486528    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.510484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.507030    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b35fbd-906b-450d-99a0-b5b0f405b607-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.526777    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.566739    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.527738    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.528192    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.528551    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.532533    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.527340    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.576570    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.576656    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.566574    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.596495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.592378    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.628483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.615734    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpxb\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-kube-api-access-nbpxb\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.662478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.637119    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:21:06.691547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.646864    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.742518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.722722    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:06.742518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.722786    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:06.742518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.725199    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:21:06.742518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.725274    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"]
Oct 30 01:21:06.742518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.731586    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.815574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.812634    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6567391a-hjzbd"]
Oct 30 01:21:06.815574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.813595    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:06.917482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.867425    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:06.917482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.906378    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:21:06.917482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.909194    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:06.917482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.909243    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:06.971346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.970485    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:21:07.001488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:06.992233    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:07.062480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.043824    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:07.121965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.115583    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:07.121965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.116609    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:07.121965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.117007    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:21:07.273722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.273675    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:07.273997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.273967    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:07.403384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.403282    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:21:07.442175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.442140    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:21:07.466874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.446028    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:21:07.466874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.451753    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:21:07.466874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.452293    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:07.466874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.464706    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:21:07.498844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.498774    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:07.499256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.499200    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:07.572854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.572800    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:21:07.636763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.636721    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:07.670782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.670738    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:07.733914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.733845    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:07.926539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.925547    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:21:07.951482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.932627    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:21:07.951482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.933013    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:21:07.951482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.933620    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:21:07.962939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.962900    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:07.963469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.963403    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:07.979507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.972589    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:07.999522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:07.997609    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:21:08.061481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.046450    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:08.061481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.046551    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:08.069669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.069616    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:08.198531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.174605    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:08.198531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.181783    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:21:08.198531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.183165    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:21:08.225339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.225273    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:08.301489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.239070    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:08.301489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.239143    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:08.301489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.297579    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:08.303031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.302106    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:21:08.345493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.323381    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:21:08.417638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.407546    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:21:08.526262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.519076    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:21:08.601494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.599589    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:08.601494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.600014    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:08.840544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.835247    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:21:08.917483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.864619    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:21:08.917483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.864995    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:21:08.917483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.905601    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:08.917483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.905665    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:08.965481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.933300    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:08.965481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.933366    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:08.965481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.934027    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:08.965481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:08.934095    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:09.003714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.003662    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:09.024764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.021569    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:09.270482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.242063    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:09.325732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.267810    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:09.352612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.352564    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:09.396638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.396589    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:09.438708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.438601    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:09.533062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.533012    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.402208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.577668    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:10.402479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.402423    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:10.403391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.604579    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.417756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.867961    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.422741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.868006    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.423140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.866667    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.423438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.070793    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:21:10.423778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.332848    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.424481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.333207    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:10.424481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.670286    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:10.424481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.424388    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:10.424481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.776875    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:10.465633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.424447    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:10.465633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.093165    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:10.465633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.424878    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:10.465633 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:10.126315    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b35fbd_906b_450d_99a0_b5b0f405b607.slice/crio-b89807a1d7fe9a68b162fb387c23b23124bab0464a683ec566d4f2599b0374f0 WatchSource:0}: Error finding container b89807a1d7fe9a68b162fb387c23b23124bab0464a683ec566d4f2599b0374f0: Status 404 returned error can't find the container with id b89807a1d7fe9a68b162fb387c23b23124bab0464a683ec566d4f2599b0374f0
Oct 30 01:21:10.465633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:09.611791    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.527608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.527566    5104 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-wfw4k container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:21:10.527923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.527889    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k" podUID="7a355cec-6892-4b36-89b7-60c6b3460bf0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.535586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.535535    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:21:10.777515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.561543    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:10.777515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.622703    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:10.777515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.765574    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:10.777515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.739147    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:10.777515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.765687    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:10.777515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.765744    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:10.777515 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:21:10.693112    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:21:10.803483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.797140    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4b7315-a1ec-4bab-bce7-96395353bd5d" path="/var/lib/kubelet/pods/6c4b7315-a1ec-4bab-bce7-96395353bd5d/volumes"
Oct 30 01:21:10.803483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.797224    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:10.803483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.797281    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:10.807532 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:21:10.806782    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.207s"
Oct 30 01:21:10.874541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.874501    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:11.094964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.082847    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:11.094964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:10.928623    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:11.094964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.082957    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:11.264482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.264146    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:11.359993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.359889    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:21:11.407909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.407841    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:11.423271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.423220    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:11.442862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.438595    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:21:11.443271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.443229    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:21:11.455612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.455568    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:11.477963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.477924    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:21:11.482751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.482726    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:21:11.498516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.456633    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:11.498835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.498800    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:11.527182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.527139    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:11.527504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.527445    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:11.551620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.551574    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:21:11.552011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.551987    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:21:11.552178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.552164    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:21:11.552311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.552299    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:21:11.552503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.552438    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:21:11.602483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:11.598385    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" event={"ID":"f8b35fbd-906b-450d-99a0-b5b0f405b607","Type":"ContainerStarted","Data":"b89807a1d7fe9a68b162fb387c23b23124bab0464a683ec566d4f2599b0374f0"}
Oct 30 01:21:12.108476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.092268    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:12.121960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.116403    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:12.121960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.117000    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:12.121960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.117043    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:12.216484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.142215    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:12.585531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.585486    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:12.586252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.585930    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:12.600531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.589975    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:12.600531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.590542    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:12.600531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.593977    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:21:12.600531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.594277    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:21:12.619815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.605570    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:21:12.619815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.606850    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:21:12.619815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.606894    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"]
Oct 30 01:21:12.619815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.610555    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:21:12.654770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.654721    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:12.682545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.671252    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:21:12.689576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.671292    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:12.690155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.679815    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"]
Oct 30 01:21:12.697162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.697138    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:21:12.817631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.679920    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:12.969625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.969568    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:12.970158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.970032    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:13.024545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:12.970511    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 01:21:13.029658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.029613    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 01:21:13.059096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.047664    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:21:13.059096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.047714    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.059096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.047733    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 30 01:21:13.059096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.049494    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:13.059096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.049540    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063238    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn79n\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-kube-api-access-pn79n\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063318    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063400    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063469    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063519    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063686    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063734    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063825    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.066496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.063870    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.129624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.129565    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.157673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.157631    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.158857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.158823    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.275127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.275032    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 01:21:13.275443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.275412    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 01:21:13.375483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.371194    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:21:13.542808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.542691    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.548147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.548111    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.548623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.548580    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn79n\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-kube-api-access-pn79n\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.556205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.555967    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.591503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.556050    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.591503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.588618    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.591503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.588668    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.591503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.588722    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.591503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.588924    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.591503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.588964    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.609884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.609846    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.610156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.553609    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 01:21:13.610305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.610282    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 01:21:13.615526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.611148    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.632652    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.633323    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.633761    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.634447    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.635414    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.660569    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.661022    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.605443    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.669356    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.716523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.669663    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.720864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.720422    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:13.749538 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:21:13.722841    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.026s"
Oct 30 01:21:13.749538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.746099    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.749538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.746795    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn79n\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-kube-api-access-pn79n\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.749538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.746856    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.749538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.748426    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:21:13.749538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.748550    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:21:13.749538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.748618    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:21:13.754242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.754215    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:21:13.766355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.763200    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:21:13.795589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.771319    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:13.795589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.772883    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:13.803369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.803334    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:21:13.873480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.859014    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:13.900479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:13.875716    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:21:14.015911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.015866    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:14.024988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.024870    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:21:14.027149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.025830    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:21:14.027562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.027529    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:21:14.027901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.027862    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:21:14.030363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.029327    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:21:14.081534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.081354    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:21:14.164749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.164721    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:21:14.240086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.240050    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:21:14.266724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.263302    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:21:14.266724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.265665    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:21:14.272976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.271677    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:21:14.276828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.276797    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:21:14.279145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.279111    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:21:14.285754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.285725    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:21:14.368277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.357419    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"]
Oct 30 01:21:14.515381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.513520    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:21:14.530302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.530227    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:21:14.621902 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:14.621825    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab5a5f5_36f7_4f21_979f_c013a3e65d2b.slice/crio-84e9f4b6a746cb86b0976baca3b3a796fd9723d28b87942ff300dddf4746167d WatchSource:0}: Error finding container 84e9f4b6a746cb86b0976baca3b3a796fd9723d28b87942ff300dddf4746167d: Status 404 returned error can't find the container with id 84e9f4b6a746cb86b0976baca3b3a796fd9723d28b87942ff300dddf4746167d
Oct 30 01:21:14.625893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.625818    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:14.626234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.626201    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:14.626326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.626281    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:14.642860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.642829    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"]
Oct 30 01:21:14.687439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.687403    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:21:14.690965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.690908    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:21:14.958501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.958432    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" event={"ID":"f8b35fbd-906b-450d-99a0-b5b0f405b607","Type":"ContainerStarted","Data":"9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d"}
Oct 30 01:21:14.965668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.965614    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" event={"ID":"dab5a5f5-36f7-4f21-979f-c013a3e65d2b","Type":"ContainerStarted","Data":"4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd"}
Oct 30 01:21:14.965965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:14.965941    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" event={"ID":"dab5a5f5-36f7-4f21-979f-c013a3e65d2b","Type":"ContainerStarted","Data":"84e9f4b6a746cb86b0976baca3b3a796fd9723d28b87942ff300dddf4746167d"}
Oct 30 01:21:15.002972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:15.002906    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podStartSLOduration=10.002887525 podStartE2EDuration="10.002887525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:14.999972869 +0000 UTC m=+2316.547663366" watchObservedRunningTime="2026-10-30 01:21:15.002887525 +0000 UTC m=+2316.550578022"
Oct 30 01:21:15.004563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:15.004530    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:15.013178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:15.013131    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:15.038200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:15.038141    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podStartSLOduration=7.038123967 podStartE2EDuration="7.038123967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:15.037760502 +0000 UTC m=+2316.585451001" watchObservedRunningTime="2026-10-30 01:21:15.038123967 +0000 UTC m=+2316.585814463"
Oct 30 01:21:15.860527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:15.860492    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:15.861774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:15.861748    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Oct 30 01:21:15.861845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:15.861800    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Oct 30 01:21:16.012755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:16.012692    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:16.014162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:16.014115    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:21:16.860652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:16.860620    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Oct 30 01:21:16.861255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:16.860678    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Oct 30 01:21:17.862517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:17.862423    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Oct 30 01:21:17.862517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:17.862503    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Oct 30 01:21:18.064422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:18.061204    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"]
Oct 30 01:21:18.164063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:18.163963    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33753a2-kbqts"]
Oct 30 01:21:18.616920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:18.616880    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63901c75-9057-45af-af45-92c74df0cbfb" path="/var/lib/kubelet/pods/63901c75-9057-45af-af45-92c74df0cbfb/volumes"
Oct 30 01:21:18.860769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:18.860719    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Oct 30 01:21:18.860989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:18.860809    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Oct 30 01:21:19.860560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:19.860522    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.861409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:19.861277    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Oct 30 01:21:20.489760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.489714    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"]
Oct 30 01:21:20.548531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.548500    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.551013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.550956    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"]
Oct 30 01:21:20.565243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.564384    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4be164-openshift-default-dockercfg-qn5hp\""
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.593935    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594003    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-envoy\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594033    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffg7\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-kube-api-access-vffg7\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594069    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70785039-29f0-4fee-ae6e-3719b53420c0-istio-podinfo\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594101    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70785039-29f0-4fee-ae6e-3719b53420c0-istiod-ca-cert\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594133    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-credential-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594158    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-certs\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594188    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-istio-token\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.594736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.594484    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-data\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.703484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.702742    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffg7\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-kube-api-access-vffg7\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.703484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.702809    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70785039-29f0-4fee-ae6e-3719b53420c0-istio-podinfo\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.703484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.702864    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70785039-29f0-4fee-ae6e-3719b53420c0-istiod-ca-cert\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.703484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.702902    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-credential-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.703484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.702926    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-certs\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.703484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.702964    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-istio-token\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.703484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.703386    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-data\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.704031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.703521    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.704031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.703577    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-envoy\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.704131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.704049    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-certs\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.704342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.704315    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-credential-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.704763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.704734    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70785039-29f0-4fee-ae6e-3719b53420c0-istiod-ca-cert\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.705016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.704997    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-data\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.705099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.705039    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.706525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.706498    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70785039-29f0-4fee-ae6e-3719b53420c0-istio-podinfo\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.707099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.707078    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-envoy\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.769542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.760952    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-istio-token\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.769542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.761708    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffg7\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-kube-api-access-vffg7\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:20.860141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.860100    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Oct 30 01:21:20.860342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.860180    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Oct 30 01:21:20.926081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:20.926037    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:21.313096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:21.313060    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"]
Oct 30 01:21:21.315237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:21.314391    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:21.315237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:21.314504    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:21.315237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:21.314540    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:21.458089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:21.457417    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" event={"ID":"70785039-29f0-4fee-ae6e-3719b53420c0","Type":"ContainerStarted","Data":"a06f4bfebb027edeacdd6a54a6a2a4e56788169e405f63c5ad285c7db6e5d275"}
Oct 30 01:21:22.032569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.032532    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Oct 30 01:21:22.033156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.033124    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Oct 30 01:21:22.468608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.468495    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" event={"ID":"70785039-29f0-4fee-ae6e-3719b53420c0","Type":"ContainerStarted","Data":"a70cc2a52a6d25ab3952f306807882193d04bac7026d931c0709e4c5531d8bec"}
Oct 30 01:21:22.524500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.522257    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podStartSLOduration=2.522232946 podStartE2EDuration="2.522232946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:22.513773212 +0000 UTC m=+2324.061463709" watchObservedRunningTime="2026-10-30 01:21:22.522232946 +0000 UTC m=+2324.069923443"
Oct 30 01:21:22.866564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.866467    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:22.866990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.866965    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:22.867784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.867763    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:21:22.926975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.926941    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:22.933353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:22.933322    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:23.485478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:23.481283    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:23.485478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:23.484676    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:21:26.958488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:26.950412    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"]
Oct 30 01:21:26.958488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:26.957483    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.021780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.015949    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4be164-dockercfg-f42nf\""
Oct 30 01:21:27.021780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.016065    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4be164\""
Oct 30 01:21:27.052522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.038507    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.052522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.038939    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db87m\" (UniqueName: \"kubernetes.io/projected/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-kube-api-access-db87m\") pod \"payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.081847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.065288    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"]
Oct 30 01:21:27.276531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.275846    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db87m\" (UniqueName: \"kubernetes.io/projected/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-kube-api-access-db87m\") pod \"payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.276531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.275923    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.283063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.276969    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.308187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.307921    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db87m\" (UniqueName: \"kubernetes.io/projected/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-kube-api-access-db87m\") pod \"payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.578592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.578507    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:27.892608 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:27.892571    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f0591a_4b9d_4c4c_9b4d_0c4b958a8353.slice/crio-8654838a97137778f2ff3292ddbd75a2a8febdacf3b5cd41eb8eceb1d3e65394 WatchSource:0}: Error finding container 8654838a97137778f2ff3292ddbd75a2a8febdacf3b5cd41eb8eceb1d3e65394: Status 404 returned error can't find the container with id 8654838a97137778f2ff3292ddbd75a2a8febdacf3b5cd41eb8eceb1d3e65394
Oct 30 01:21:27.893512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:27.893481    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"]
Oct 30 01:21:28.541852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:28.541810    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk" event={"ID":"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353","Type":"ContainerStarted","Data":"4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f"}
Oct 30 01:21:28.542258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:28.541877    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk" event={"ID":"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353","Type":"ContainerStarted","Data":"8654838a97137778f2ff3292ddbd75a2a8febdacf3b5cd41eb8eceb1d3e65394"}
Oct 30 01:21:28.542258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:28.541908    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:28.579903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:28.579844    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk" podStartSLOduration=2.579828413 podStartE2EDuration="2.579828413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:28.577830854 +0000 UTC m=+2330.125521351" watchObservedRunningTime="2026-10-30 01:21:28.579828413 +0000 UTC m=+2330.127518974"
Oct 30 01:21:34.529451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:34.529411    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:21:36.111479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:36.107515    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"]
Oct 30 01:21:36.142480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:36.139161    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c294ca-hjjq7"]
Oct 30 01:21:36.738962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:36.738921    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a35d71-6e8d-4846-b7d1-c8008ebe9657" path="/var/lib/kubelet/pods/07a35d71-6e8d-4846-b7d1-c8008ebe9657/volumes"
Oct 30 01:21:39.252834 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:21:39.252782    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:21:39.557662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:39.557577    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:21:40.042819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:40.042778    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"]
Oct 30 01:21:40.049289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:40.049253    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fca894e-q85q5"]
Oct 30 01:21:40.616161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:40.616099    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7537d8-e5c4-454a-b10c-a7c035d1cb22" path="/var/lib/kubelet/pods/cf7537d8-e5c4-454a-b10c-a7c035d1cb22/volumes"
Oct 30 01:21:46.029733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:46.029681    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"]
Oct 30 01:21:46.041350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:46.041311    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd9d8d0-66wqr"]
Oct 30 01:21:46.587384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:46.587345    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f894045c-f5a9-4272-9cba-bb7f6156fc19" path="/var/lib/kubelet/pods/f894045c-f5a9-4272-9cba-bb7f6156fc19/volumes"
Oct 30 01:21:47.219103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.219052    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"]
Oct 30 01:21:47.227556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.227524    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.269845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.269808    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"]
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.349556    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.349627    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.349672    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.349745    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.349836    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccr2g\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-kube-api-access-ccr2g\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.350002    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.350080    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.350103    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092373c6-de73-4075-9c94-340796cb9455-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.395227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.350140    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092373c6-de73-4075-9c94-340796cb9455-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.452349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.452304    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.452756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.452729    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccr2g\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-kube-api-access-ccr2g\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.452988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.452970    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.453160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.453146    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.453279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.453265    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092373c6-de73-4075-9c94-340796cb9455-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.453411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.453395    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092373c6-de73-4075-9c94-340796cb9455-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.453709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.453686    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.453863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.453847    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.454013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.453999    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.454175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.454145    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092373c6-de73-4075-9c94-340796cb9455-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.454249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.453889    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.454249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.454213    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.454377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.454050    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.454854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.454831    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.456122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.456092    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.457282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.457257    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092373c6-de73-4075-9c94-340796cb9455-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.470925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.470849    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccr2g\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-kube-api-access-ccr2g\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.471070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.471043    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.576229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.576195    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:47.945774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.945733    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"]
Oct 30 01:21:47.946032 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:47.946006    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092373c6_de73_4075_9c94_340796cb9455.slice/crio-726085de7ea523cfbec2cdeb55890636578c69ec671ce4e675253f47fb11fa96 WatchSource:0}: Error finding container 726085de7ea523cfbec2cdeb55890636578c69ec671ce4e675253f47fb11fa96: Status 404 returned error can't find the container with id 726085de7ea523cfbec2cdeb55890636578c69ec671ce4e675253f47fb11fa96
Oct 30 01:21:47.949812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.949772    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:47.949938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.949863    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:47.949938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:47.949911    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:48.138276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:48.135763    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" event={"ID":"092373c6-de73-4075-9c94-340796cb9455","Type":"ContainerStarted","Data":"b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"}
Oct 30 01:21:48.138276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:48.135835    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" event={"ID":"092373c6-de73-4075-9c94-340796cb9455","Type":"ContainerStarted","Data":"726085de7ea523cfbec2cdeb55890636578c69ec671ce4e675253f47fb11fa96"}
Oct 30 01:21:48.178435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:48.178366    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podStartSLOduration=1.178346248 podStartE2EDuration="1.178346248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:48.176852638 +0000 UTC m=+2349.724543137" watchObservedRunningTime="2026-10-30 01:21:48.178346248 +0000 UTC m=+2349.726036744"
Oct 30 01:21:48.585603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:48.585561    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:49.150341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.150306    5104 generic.go:356] "Generic (PLEG): container finished" podID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerID="f3631caf395c09e7e20feaf56189b9c3db723ec3bfe09a8007ebcc93e2ce279f" exitCode=0
Oct 30 01:21:49.150569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.150379    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" event={"ID":"1d563dff-75e3-48e2-b6ca-4dd41a6cca56","Type":"ContainerDied","Data":"f3631caf395c09e7e20feaf56189b9c3db723ec3bfe09a8007ebcc93e2ce279f"}
Oct 30 01:21:49.151024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.151004    5104 scope.go:122] "RemoveContainer" containerID="f3631caf395c09e7e20feaf56189b9c3db723ec3bfe09a8007ebcc93e2ce279f"
Oct 30 01:21:49.152913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.152883    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:49.153028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.152957    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:49.153028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.152992    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:49.579537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.579137    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 01:21:49.579537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:49.579205    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 01:21:50.028486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.028418    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"]
Oct 30 01:21:50.033507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.033481    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f821dd22-mblzg"]
Oct 30 01:21:50.162550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.162519    5104 generic.go:356] "Generic (PLEG): container finished" podID="c888e628-a915-4c08-b192-10976b6148cc" containerID="f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5" exitCode=0
Oct 30 01:21:50.162769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.162599    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" event={"ID":"c888e628-a915-4c08-b192-10976b6148cc","Type":"ContainerDied","Data":"f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5"}
Oct 30 01:21:50.163214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.163184    5104 scope.go:122] "RemoveContainer" containerID="f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5"
Oct 30 01:21:50.165434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.165401    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" event={"ID":"1d563dff-75e3-48e2-b6ca-4dd41a6cca56","Type":"ContainerStarted","Data":"79d66a9f60797ea53ea5f82191493a404fae9ce49790edd55aab8fa9251aab79"}
Oct 30 01:21:50.165635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.165604    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:50.165729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.165687    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:50.165800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.165737    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:50.454546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.424263    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:21:50.454546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.424388    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:50.454546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.424443    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:50.608486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.577291    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 01:21:50.608486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.577440    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 01:21:50.639483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:50.639312    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850c8723-4307-44b1-b6e0-91e76568f158" path="/var/lib/kubelet/pods/850c8723-4307-44b1-b6e0-91e76568f158/volumes"
Oct 30 01:21:51.217543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.217500    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" event={"ID":"c888e628-a915-4c08-b192-10976b6148cc","Type":"ContainerStarted","Data":"f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8"}
Oct 30 01:21:51.342482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.341486    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:21:51.342482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.341633    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:51.342482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.341701    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:51.409321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.409281    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:51.410084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.410049    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:51.580495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.577888    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 01:21:51.580495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:51.577971    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 01:21:52.342481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:52.341912    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:52.342481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:52.341990    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:52.409481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:52.409365    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:52.409481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:52.409425    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:52.593576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:52.578286    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 01:21:52.593576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:52.578354    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 01:21:53.341482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.334018    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:53.341482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.334094    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:53.411482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.409643    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:53.411482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.409709    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:53.602538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.595146    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 01:21:53.602538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.595204    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 01:21:53.632488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.631497    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"]
Oct 30 01:21:53.714616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.708812    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.923956    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924088    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-kube-api-access-px9q4\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924139    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924211    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924284    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924365    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924420    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924540    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:53.927522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.924593    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.044744    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.044880    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-kube-api-access-px9q4\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.044937    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.045010    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.045098    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.045171    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.045223    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.045316    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.045370    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.046217    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.148233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.053962    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.148233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.054996    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.148233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:53.789443    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"]
Oct 30 01:21:54.148233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.074318    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.185176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.185067    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.253424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.245040    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:21:54.329302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.329244    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:54.393581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.335124    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.393581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.342951    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.456923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.439623    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:21:54.456923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.446565    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:54.456923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.446630    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:54.456923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.446682    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:54.456923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.446729    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:54.468013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.467928    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-kube-api-access-px9q4\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.495769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.495718    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:54.516483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.515255    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.543394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.536346    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:54.580491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.578351    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 01:21:54.580491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.578410    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 01:21:54.686622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.686273    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:54.903481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.900750    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:21:54.931332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.931280    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:54.992492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:54.931877    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.061838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.040181    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.089916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.089867    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:21:55.233207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.233168    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.258534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.245627    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.258534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.246508    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:21:55.344541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.341711    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.344541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.342163    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:55.409112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.376157    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:55.409112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.376240    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:55.441498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.437318    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:55.441498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.437399    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:55.451232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.451187    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:21:55.509490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.468620    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:21:55.509490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.468755    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:21:55.509490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.468782    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:21:55.534595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.534548    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.535870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.535840    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:21:55.747581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.562930    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.767926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.763496    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:21:55.770252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.634623    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 01:21:55.770559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.770524    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 01:21:55.770744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.581901    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.951589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.951544    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 01:21:55.958755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.951862    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:55.991524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:55.962584    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:21:56.137108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.137070    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:21:56.155763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.155712    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:56.164483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.158576    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:21:56.164483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.160947    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:21:56.164483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.161963    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:21:56.193239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.193204    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:21:56.273363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.273316    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:56.315928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.315892    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:21:56.316625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.316604    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:21:56.316825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.316812    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:21:56.338980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.338949    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:21:56.344489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.343544    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:56.353762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.347650    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:56.353762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.347717    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:56.354376 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:21:56.354348    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ac2245_ed16_4744_b7c8_cc67c20b47b7.slice/crio-3268358efa9c7d74ac22c4a51ef33e5bfc88b3f4a29612bc4e292211ac0742db WatchSource:0}: Error finding container 3268358efa9c7d74ac22c4a51ef33e5bfc88b3f4a29612bc4e292211ac0742db: Status 404 returned error can't find the container with id 3268358efa9c7d74ac22c4a51ef33e5bfc88b3f4a29612bc4e292211ac0742db
Oct 30 01:21:56.378147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.370266    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:21:56.411096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.411059    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:56.411398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.411361    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:56.416067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.415348    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:56.416668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.416638    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:56.418308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.418280    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:21:56.438715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.438066    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"]
Oct 30 01:21:56.465572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.465529    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:56.467091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.465945    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:21:56.562613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.556365    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:21:56.913498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.913388    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:21:56.937138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.933634    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:56.938914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.938489    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:21:56.939416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.939396    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:56.962171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:56.962140    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:21:57.069631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.069589    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"]
Oct 30 01:21:57.149542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.146157    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb5c7fe0-hwllp"]
Oct 30 01:21:57.162976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.162921    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" event={"ID":"e4ac2245-ed16-4744-b7c8-cc67c20b47b7","Type":"ContainerStarted","Data":"d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9"}
Oct 30 01:21:57.163208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.162994    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" event={"ID":"e4ac2245-ed16-4744-b7c8-cc67c20b47b7","Type":"ContainerStarted","Data":"3268358efa9c7d74ac22c4a51ef33e5bfc88b3f4a29612bc4e292211ac0742db"}
Oct 30 01:21:57.352362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.352323    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:57.355624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.355584    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:57.449036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.448993    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 01:21:57.449448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.449342    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 01:21:57.578482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.543642    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podStartSLOduration=4.543617512 podStartE2EDuration="4.543617512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:57.49005454 +0000 UTC m=+2359.037745038" watchObservedRunningTime="2026-10-30 01:21:57.543617512 +0000 UTC m=+2359.091308000"
Oct 30 01:21:57.749484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.709842    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:21:57.763528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.750955    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:21:57.815896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:57.815846    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:21:58.338975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:58.338918    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 01:21:58.339952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:58.339906    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 01:21:58.441261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:58.440802    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:21:58.442262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:58.442240    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:21:58.722728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:58.711056    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:21:58.722728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:58.711120    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:21:58.725046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:58.725011    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f6822e-b4ee-48ec-9d64-5972ee382a68" path="/var/lib/kubelet/pods/55f6822e-b4ee-48ec-9d64-5972ee382a68/volumes"
Oct 30 01:21:59.039772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:59.039663    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"]
Oct 30 01:21:59.044816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:59.044776    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9066fe-9r86c"]
Oct 30 01:21:59.422347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:59.421220    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:21:59.442374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:59.442331    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:21:59.759119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:59.758918    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:21:59.760345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:21:59.759828    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:22:00.119776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.119668    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"]
Oct 30 01:22:00.149848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.149817    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"]
Oct 30 01:22:00.150062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.149986    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.164798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.164748    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b2a05c-openshift-default-dockercfg-lgjtt\""
Oct 30 01:22:00.666518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.658506    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.667089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.667057    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.667286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.667262    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.667503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.667482    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2jg\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-kube-api-access-pb2jg\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.667706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.667682    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.667835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.667820    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.668043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.668008    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.668194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.668177    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.668335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.668320    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.728893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.728847    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:22:00.729056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.728935    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:22:00.753302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.753261    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4" path="/var/lib/kubelet/pods/4e6265cd-38aa-4257-bb7a-8f2e11fd8fa4/volumes"
Oct 30 01:22:00.770384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.770309    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.770887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.770420    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.770887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.770500    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.770887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.770874    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771016    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771120    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771187    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771212    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2jg\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-kube-api-access-pb2jg\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771273    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771310    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771404    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771619    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.771759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.771740    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.772279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.772252    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.790636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.790572    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.802634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.791542    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.811496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.808167    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.815677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.815644    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2jg\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-kube-api-access-pb2jg\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:00.962262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.962224    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"]
Oct 30 01:22:00.995483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:00.993677    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"]
Oct 30 01:22:01.087285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:01.085753    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:01.705483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:01.704890    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:22:01.705483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:01.704951    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:22:01.758635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:01.756120    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"]
Oct 30 01:22:01.786003 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:01.777820    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cf53c2_ab95_4e5a_a404_4ed17a140a1c.slice/crio-d4d30e657e3aab6506c1701f701f27a5c4449c4c16d6a37c16badcf9ee535ae6 WatchSource:0}: Error finding container d4d30e657e3aab6506c1701f701f27a5c4449c4c16d6a37c16badcf9ee535ae6: Status 404 returned error can't find the container with id d4d30e657e3aab6506c1701f701f27a5c4449c4c16d6a37c16badcf9ee535ae6
Oct 30 01:22:01.794406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:01.794348    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:01.794877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:01.794841    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:01.795096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:01.795074    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:02.256535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.255136    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"]
Oct 30 01:22:02.264502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.263335    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.313951    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.314066    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.314098    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.314151    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316d84a-4399-420a-82cf-74c36d1a431f-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.314186    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.314243    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.314306    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316d84a-4399-420a-82cf-74c36d1a431f-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.314482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.314439    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tnt\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-kube-api-access-59tnt\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.330644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.330604    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.335126 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.335102    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"]
Oct 30 01:22:02.434794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.434747    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.435394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.435364    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.436034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.435948    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.436165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.436151    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316d84a-4399-420a-82cf-74c36d1a431f-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.436645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.436630    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.437039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.436929    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.437174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.437156    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.437326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.437306    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.437426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.437413    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316d84a-4399-420a-82cf-74c36d1a431f-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.437724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.437704    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tnt\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-kube-api-access-59tnt\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.438141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.438117    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316d84a-4399-420a-82cf-74c36d1a431f-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.438141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.438121    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.438515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.438501    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.438914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.438617    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.441602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.441583    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.459991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.458484    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.459991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.458963    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tnt\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-kube-api-access-59tnt\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.459991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.458975    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316d84a-4399-420a-82cf-74c36d1a431f-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.642756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.640122    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" event={"ID":"75cf53c2-ab95-4e5a-a404-4ed17a140a1c","Type":"ContainerStarted","Data":"3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e"}
Oct 30 01:22:02.642756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.640198    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" event={"ID":"75cf53c2-ab95-4e5a-a404-4ed17a140a1c","Type":"ContainerStarted","Data":"d4d30e657e3aab6506c1701f701f27a5c4449c4c16d6a37c16badcf9ee535ae6"}
Oct 30 01:22:02.654503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.654469    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:02.699557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.699512    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:22:02.724543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.699821    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:22:02.736529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:02.736399    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podStartSLOduration=2.736377923 podStartE2EDuration="2.736377923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:02.721436322 +0000 UTC m=+2364.269126813" watchObservedRunningTime="2026-10-30 01:22:02.736377923 +0000 UTC m=+2364.284068414"
Oct 30 01:22:03.104503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.103377    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:03.136947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.113937    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:03.307550 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:03.307517    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda316d84a_4399_420a_82cf_74c36d1a431f.slice/crio-d7ce94e330d39a2a516f648266aa9fd0158d50368c2fd026067a19dcf80f6728 WatchSource:0}: Error finding container d7ce94e330d39a2a516f648266aa9fd0158d50368c2fd026067a19dcf80f6728: Status 404 returned error can't find the container with id d7ce94e330d39a2a516f648266aa9fd0158d50368c2fd026067a19dcf80f6728
Oct 30 01:22:03.309264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.309238    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"]
Oct 30 01:22:03.310497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.310444    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:03.310597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.310551    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:03.310668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.310613    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:03.657024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.656897    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" event={"ID":"a316d84a-4399-420a-82cf-74c36d1a431f","Type":"ContainerStarted","Data":"1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5"}
Oct 30 01:22:03.657024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.656966    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" event={"ID":"a316d84a-4399-420a-82cf-74c36d1a431f","Type":"ContainerStarted","Data":"d7ce94e330d39a2a516f648266aa9fd0158d50368c2fd026067a19dcf80f6728"}
Oct 30 01:22:03.657488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.657442    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:03.658841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.658813    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:22:03.692721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.692665    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:22:03.693045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.692999    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:22:03.751559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:03.751498    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podStartSLOduration=1.751481182 podStartE2EDuration="1.751481182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:03.711206837 +0000 UTC m=+2365.258897336" watchObservedRunningTime="2026-10-30 01:22:03.751481182 +0000 UTC m=+2365.299171672"
Oct 30 01:22:04.665480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:04.659440    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:04.678347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:04.678291    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 01:22:04.678691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:04.678640    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 01:22:04.715095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:04.714643    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 30 01:22:04.715095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:04.714714    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 30 01:22:05.655478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:05.655428    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 01:22:05.655646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:05.655500    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 01:22:05.717480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:05.715517    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:22:05.717480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:05.717301    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:22:05.722491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:05.718660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:22:05.962786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:05.962732    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc" podUID="00b50f17-564f-409d-a863-2052bb8a6de3" containerName="payload-processing" containerID="cri-o://80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16" gracePeriod=25
Oct 30 01:22:06.002134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.000947    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm" podUID="8a5c39b0-ada0-4740-b2db-0f17a8f557f8" containerName="payload-pre-processing" containerID="cri-o://d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12" gracePeriod=25
Oct 30 01:22:06.411720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.411680    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:22:06.538482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.525675    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:22:06.614658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.601129    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b50f17-564f-409d-a863-2052bb8a6de3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b50f17-564f-409d-a863-2052bb8a6de3-plugins-config-volume\") pod \"00b50f17-564f-409d-a863-2052bb8a6de3\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") "
Oct 30 01:22:06.614658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.601494    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b50f17-564f-409d-a863-2052bb8a6de3-kube-api-access-7576h\" (UniqueName: \"kubernetes.io/projected/00b50f17-564f-409d-a863-2052bb8a6de3-kube-api-access-7576h\") pod \"00b50f17-564f-409d-a863-2052bb8a6de3\" (UID: \"00b50f17-564f-409d-a863-2052bb8a6de3\") "
Oct 30 01:22:06.647110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.647069    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b50f17-564f-409d-a863-2052bb8a6de3-plugins-config-volume" pod "00b50f17-564f-409d-a863-2052bb8a6de3" (UID: "00b50f17-564f-409d-a863-2052bb8a6de3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:06.654422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.654383    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b50f17-564f-409d-a863-2052bb8a6de3-kube-api-access-7576h" pod "00b50f17-564f-409d-a863-2052bb8a6de3" (UID: "00b50f17-564f-409d-a863-2052bb8a6de3"). InnerVolumeSpecName "kube-api-access-7576h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:06.655238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.655210    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 01:22:06.655373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.655260    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 01:22:06.750014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.749978    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-plugins-config-volume\") pod \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") "
Oct 30 01:22:06.750687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.750345    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-kube-api-access-lxshc\" (UniqueName: \"kubernetes.io/projected/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-kube-api-access-lxshc\") pod \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\" (UID: \"8a5c39b0-ada0-4740-b2db-0f17a8f557f8\") "
Oct 30 01:22:06.750687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.750406    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-plugins-config-volume" pod "8a5c39b0-ada0-4740-b2db-0f17a8f557f8" (UID: "8a5c39b0-ada0-4740-b2db-0f17a8f557f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:06.751334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.751307    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b50f17-564f-409d-a863-2052bb8a6de3-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:06.751518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.751335    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:06.751518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.751351    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7576h\" (UniqueName: \"kubernetes.io/projected/00b50f17-564f-409d-a863-2052bb8a6de3-kube-api-access-7576h\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:06.753147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.753121    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-kube-api-access-lxshc" pod "8a5c39b0-ada0-4740-b2db-0f17a8f557f8" (UID: "8a5c39b0-ada0-4740-b2db-0f17a8f557f8"). InnerVolumeSpecName "kube-api-access-lxshc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:06.824780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.822613    5104 generic.go:356] "Generic (PLEG): container finished" podID="00b50f17-564f-409d-a863-2052bb8a6de3" containerID="80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16" exitCode=0
Oct 30 01:22:06.824780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.822688    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"
Oct 30 01:22:06.824780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.822735    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc" event={"ID":"00b50f17-564f-409d-a863-2052bb8a6de3","Type":"ContainerDied","Data":"80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16"}
Oct 30 01:22:06.824780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.822814    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc" event={"ID":"00b50f17-564f-409d-a863-2052bb8a6de3","Type":"ContainerDied","Data":"709a4a991df87c9c669eeb899c77f0d4250df23392a8ef788252d8f2ffc94b4d"}
Oct 30 01:22:06.824780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.822843    5104 scope.go:122] "RemoveContainer" containerID="80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16"
Oct 30 01:22:06.828565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.828536    5104 generic.go:356] "Generic (PLEG): container finished" podID="8a5c39b0-ada0-4740-b2db-0f17a8f557f8" containerID="d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12" exitCode=0
Oct 30 01:22:06.828732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.828638    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"
Oct 30 01:22:06.828906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.828664    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm" event={"ID":"8a5c39b0-ada0-4740-b2db-0f17a8f557f8","Type":"ContainerDied","Data":"d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12"}
Oct 30 01:22:06.829054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.829038    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm" event={"ID":"8a5c39b0-ada0-4740-b2db-0f17a8f557f8","Type":"ContainerDied","Data":"7edcbe212efcac73a07a008ab6036f067a453bc7ec9693b10f98850301d15214"}
Oct 30 01:22:06.845041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.845016    5104 scope.go:122] "RemoveContainer" containerID="80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16"
Oct 30 01:22:06.845420 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:22:06.845391    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16\": container with ID starting with 80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16 not found: ID does not exist" containerID="80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16"
Oct 30 01:22:06.845513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.845435    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16"} err="failed to get container status \"80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16\": rpc error: code = NotFound desc = could not find container \"80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16\": container with ID starting with 80a471d0ff25a6cad3e0d152761efe2c6929195c52333b9e056377590e688f16 not found: ID does not exist"
Oct 30 01:22:06.845588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.845511    5104 scope.go:122] "RemoveContainer" containerID="d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12"
Oct 30 01:22:06.868648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.852988    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxshc\" (UniqueName: \"kubernetes.io/projected/8a5c39b0-ada0-4740-b2db-0f17a8f557f8-kube-api-access-lxshc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:06.912659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.912628    5104 scope.go:122] "RemoveContainer" containerID="d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12"
Oct 30 01:22:06.921629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.921586    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"]
Oct 30 01:22:06.925256 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:22:06.925215    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12\": container with ID starting with d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12 not found: ID does not exist" containerID="d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12"
Oct 30 01:22:06.925410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.925270    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12"} err="failed to get container status \"d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12\": rpc error: code = NotFound desc = could not find container \"d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12\": container with ID starting with d0e296c10322d77b3a8efb62a5db7f67681dfe6160a9f841f665434f4f248c12 not found: ID does not exist"
Oct 30 01:22:06.934533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.928753    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-58d9d7bc8b-wzfkc"]
Oct 30 01:22:06.969782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:06.969733    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"]
Oct 30 01:22:07.006931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.006892    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-6c67d697c5-fpvjm"]
Oct 30 01:22:07.106355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.106316    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"]
Oct 30 01:22:07.107668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.107646    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8a5c39b0-ada0-4740-b2db-0f17a8f557f8" containerName="payload-pre-processing"
Oct 30 01:22:07.107770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.107669    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5c39b0-ada0-4740-b2db-0f17a8f557f8" containerName="payload-pre-processing"
Oct 30 01:22:07.107770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.107704    5104 cpu_manager.go:414] "containerMap: removing container" podUID="00b50f17-564f-409d-a863-2052bb8a6de3" containerName="payload-processing"
Oct 30 01:22:07.107770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.107710    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b50f17-564f-409d-a863-2052bb8a6de3" containerName="payload-processing"
Oct 30 01:22:07.107914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.107903    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5c39b0-ada0-4740-b2db-0f17a8f557f8" containerName="payload-pre-processing"
Oct 30 01:22:07.107948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.107917    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b50f17-564f-409d-a863-2052bb8a6de3" containerName="payload-processing"
Oct 30 01:22:07.115215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.115169    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.143511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.137697    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2a05c-dockercfg-lwmtn\""
Oct 30 01:22:07.143972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.143944    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2a05c\""
Oct 30 01:22:07.177701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.171801    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/559b9e48-a828-4da7-807b-031b99769899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.177701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.171891    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j568g\" (UniqueName: \"kubernetes.io/projected/559b9e48-a828-4da7-807b-031b99769899-kube-api-access-j568g\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.194479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.191134    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"]
Oct 30 01:22:07.292211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.286786    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/559b9e48-a828-4da7-807b-031b99769899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.292211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.286878    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j568g\" (UniqueName: \"kubernetes.io/projected/559b9e48-a828-4da7-807b-031b99769899-kube-api-access-j568g\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.362479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.360649    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/559b9e48-a828-4da7-807b-031b99769899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.434121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.434040    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j568g\" (UniqueName: \"kubernetes.io/projected/559b9e48-a828-4da7-807b-031b99769899-kube-api-access-j568g\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.471124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.469909    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"]
Oct 30 01:22:07.567592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.559800    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:07.648531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.645849    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:07.652493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.652437    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"]
Oct 30 01:22:07.655543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.655513    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 01:22:07.655895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.655577    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 01:22:07.659815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.659773    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add97fd5-9780-47df-bb5f-c523d9700c1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:07.660191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.660151    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxtd\" (UniqueName: \"kubernetes.io/projected/add97fd5-9780-47df-bb5f-c523d9700c1a-kube-api-access-4bxtd\") pod \"payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:07.762864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.762260    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxtd\" (UniqueName: \"kubernetes.io/projected/add97fd5-9780-47df-bb5f-c523d9700c1a-kube-api-access-4bxtd\") pod \"payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:07.762864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.762435    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add97fd5-9780-47df-bb5f-c523d9700c1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:07.763656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.763630    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add97fd5-9780-47df-bb5f-c523d9700c1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:07.814841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.814767    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxtd\" (UniqueName: \"kubernetes.io/projected/add97fd5-9780-47df-bb5f-c523d9700c1a-kube-api-access-4bxtd\") pod \"payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:08.015045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:07.992652    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"]
Oct 30 01:22:08.045810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.045595    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:08.620038 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:08.619999    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd97fd5_9780_47df_bb5f_c523d9700c1a.slice/crio-87a47f179352a12f920f41876da08bc173565dddedaaa49c75092c57873deca2 WatchSource:0}: Error finding container 87a47f179352a12f920f41876da08bc173565dddedaaa49c75092c57873deca2: Status 404 returned error can't find the container with id 87a47f179352a12f920f41876da08bc173565dddedaaa49c75092c57873deca2
Oct 30 01:22:08.633921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.633889    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b50f17-564f-409d-a863-2052bb8a6de3" path="/var/lib/kubelet/pods/00b50f17-564f-409d-a863-2052bb8a6de3/volumes"
Oct 30 01:22:08.634427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.634407    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5c39b0-ada0-4740-b2db-0f17a8f557f8" path="/var/lib/kubelet/pods/8a5c39b0-ada0-4740-b2db-0f17a8f557f8/volumes"
Oct 30 01:22:08.634837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.634821    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"]
Oct 30 01:22:08.656671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.656642    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 01:22:08.656783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.656692    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 01:22:08.858506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.858420    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt" event={"ID":"add97fd5-9780-47df-bb5f-c523d9700c1a","Type":"ContainerStarted","Data":"5356b7dd1e12fdd46191562ac6673fa73e56942f1134fb485051802cc4edf0f1"}
Oct 30 01:22:08.859028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.858523    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt" event={"ID":"add97fd5-9780-47df-bb5f-c523d9700c1a","Type":"ContainerStarted","Data":"87a47f179352a12f920f41876da08bc173565dddedaaa49c75092c57873deca2"}
Oct 30 01:22:08.859028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.858558    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:08.860659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.860627    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m" event={"ID":"559b9e48-a828-4da7-807b-031b99769899","Type":"ContainerStarted","Data":"73df0d98d557de28c1a850a34883a08b5e6080f65fcedea61c60a3c7bb2b2fb4"}
Oct 30 01:22:08.860811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.860686    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m" event={"ID":"559b9e48-a828-4da7-807b-031b99769899","Type":"ContainerStarted","Data":"a9437db512bee6de7306faf20fc538d6117226cf7cef795108220a34bf2f327c"}
Oct 30 01:22:08.860811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.860716    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:08.905720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.897741    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt" podStartSLOduration=1.897721154 podStartE2EDuration="1.897721154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:08.896927477 +0000 UTC m=+2370.444617975" watchObservedRunningTime="2026-10-30 01:22:08.897721154 +0000 UTC m=+2370.445411653"
Oct 30 01:22:08.937173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:08.937106    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m" podStartSLOduration=1.9370869659999999 podStartE2EDuration="1.937086966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:08.932696561 +0000 UTC m=+2370.480387057" watchObservedRunningTime="2026-10-30 01:22:08.937086966 +0000 UTC m=+2370.484777463"
Oct 30 01:22:09.280746 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:22:09.280651    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:22:09.656155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:09.656066    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 01:22:09.656155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:09.656123    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 01:22:10.029060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:10.029020    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"]
Oct 30 01:22:10.042267 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:10.042226    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cea6f9-k2x87"]
Oct 30 01:22:10.581126 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:10.581093    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d148f0d-2186-472c-81df-0d88cef41a48" path="/var/lib/kubelet/pods/6d148f0d-2186-472c-81df-0d88cef41a48/volumes"
Oct 30 01:22:10.660876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:10.660847    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:10.661332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:10.661314    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:10.662037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:10.662010    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:22:19.943906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:19.943812    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:22:19.944378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:19.943961    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:22:21.896871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:21.896829    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:22:24.876492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:24.873848    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:25.256480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.256078    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:22:25.287743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.258146    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:22:25.379480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.366141    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:22:25.507539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.460690    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:22:25.566518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.553732    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:25.566518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.493385    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:22:25.566518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.493097    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:22:25.566518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.537574    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:25.604481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.589727    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:22:25.987159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:25.981531    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.027748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.027710    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:22:26.096556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.096443    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.195621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.195566    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.352610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.346120    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.364493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.353403    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:22:26.386496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.366654    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.386496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.367109    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:22:26.540933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.540874    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:22:26.540933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.541660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:22:26.553988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.553944    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.555853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.555046    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:22:26.561380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.560567    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.561380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.561445    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.561380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.563781    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:22:26.570968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.570660    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" probeResult="failure" output="Get \"https://10.132.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.582163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.571056    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:22:26.582163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.571590    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.596483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.595582    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:26.993718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:26.988070    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:27.343714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.343596    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419008    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419115    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419166    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419203    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419249    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419282    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419330    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:22:27.419478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.419365    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:22:27.464524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.464488    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:22:27.506359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.495493    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"]
Oct 30 01:22:27.546199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:27.546167    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbba2121-zhqhn"]
Oct 30 01:22:28.066308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.066269    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"]
Oct 30 01:22:28.075575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.075541    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.110533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.107507    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:22:28.110533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.107610    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:22:28.110533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.107636    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:28.134486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.128212    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:22:28.171948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.171819    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"]
Oct 30 01:22:28.241496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.238244    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.241496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.238323    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnfv\" (UniqueName: \"kubernetes.io/projected/989267ea-10a8-4299-a24e-ee194aa3616e-kube-api-access-rtnfv\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.241496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.238496    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.348555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.345186    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.348555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.345260    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnfv\" (UniqueName: \"kubernetes.io/projected/989267ea-10a8-4299-a24e-ee194aa3616e-kube-api-access-rtnfv\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.348555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.345376    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.348555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.346482    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.359838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.359803    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.408394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.408354    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnfv\" (UniqueName: \"kubernetes.io/projected/989267ea-10a8-4299-a24e-ee194aa3616e-kube-api-access-rtnfv\") pod \"maas-api-revoke-sub-6d0fed33-nmlb6\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.569641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.569604    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:28.629085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.629045    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208f7391-3f08-434e-aa58-8652e6e34ac0" path="/var/lib/kubelet/pods/208f7391-3f08-434e-aa58-8652e6e34ac0/volumes"
Oct 30 01:22:28.958839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:28.958799    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"]
Oct 30 01:22:28.960064 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:28.960037    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989267ea_10a8_4299_a24e_ee194aa3616e.slice/crio-a027b0acae2fefe9a2805c700cdeca191acf0bc5f2809eb8a2c0b31afa5c0749 WatchSource:0}: Error finding container a027b0acae2fefe9a2805c700cdeca191acf0bc5f2809eb8a2c0b31afa5c0749: Status 404 returned error can't find the container with id a027b0acae2fefe9a2805c700cdeca191acf0bc5f2809eb8a2c0b31afa5c0749
Oct 30 01:22:29.351608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:29.351519    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6" event={"ID":"989267ea-10a8-4299-a24e-ee194aa3616e","Type":"ContainerStarted","Data":"a027b0acae2fefe9a2805c700cdeca191acf0bc5f2809eb8a2c0b31afa5c0749"}
Oct 30 01:22:30.366189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:30.366148    5104 generic.go:356] "Generic (PLEG): container finished" podID="989267ea-10a8-4299-a24e-ee194aa3616e" containerID="381db6d276acd3ba761d343e6ec7272f2817124e61ac7d2b96dfc3bbf083d852" exitCode=0
Oct 30 01:22:30.366641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:30.366243    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6" event={"ID":"989267ea-10a8-4299-a24e-ee194aa3616e","Type":"ContainerDied","Data":"381db6d276acd3ba761d343e6ec7272f2817124e61ac7d2b96dfc3bbf083d852"}
Oct 30 01:22:31.795482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.785698    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:31.835946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.834447    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-openshift-ca\") pod \"989267ea-10a8-4299-a24e-ee194aa3616e\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") "
Oct 30 01:22:31.835946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.834644    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989267ea-10a8-4299-a24e-ee194aa3616e-kube-api-access-rtnfv\" (UniqueName: \"kubernetes.io/projected/989267ea-10a8-4299-a24e-ee194aa3616e-kube-api-access-rtnfv\") pod \"989267ea-10a8-4299-a24e-ee194aa3616e\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") "
Oct 30 01:22:31.835946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.834853    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-xks-ca\") pod \"989267ea-10a8-4299-a24e-ee194aa3616e\" (UID: \"989267ea-10a8-4299-a24e-ee194aa3616e\") "
Oct 30 01:22:31.835946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.835021    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-openshift-ca" pod "989267ea-10a8-4299-a24e-ee194aa3616e" (UID: "989267ea-10a8-4299-a24e-ee194aa3616e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:31.835946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.835892    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:31.839808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.839766    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-xks-ca" pod "989267ea-10a8-4299-a24e-ee194aa3616e" (UID: "989267ea-10a8-4299-a24e-ee194aa3616e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:31.840369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.840341    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989267ea-10a8-4299-a24e-ee194aa3616e-kube-api-access-rtnfv" pod "989267ea-10a8-4299-a24e-ee194aa3616e" (UID: "989267ea-10a8-4299-a24e-ee194aa3616e"). InnerVolumeSpecName "kube-api-access-rtnfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:31.937663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.937629    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtnfv\" (UniqueName: \"kubernetes.io/projected/989267ea-10a8-4299-a24e-ee194aa3616e-kube-api-access-rtnfv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:31.937663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:31.937661    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989267ea-10a8-4299-a24e-ee194aa3616e-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:32.420811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.420775    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"
Oct 30 01:22:32.421379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.421342    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6" event={"ID":"989267ea-10a8-4299-a24e-ee194aa3616e","Type":"ContainerDied","Data":"a027b0acae2fefe9a2805c700cdeca191acf0bc5f2809eb8a2c0b31afa5c0749"}
Oct 30 01:22:32.421657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.421631    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a027b0acae2fefe9a2805c700cdeca191acf0bc5f2809eb8a2c0b31afa5c0749"
Oct 30 01:22:32.510959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.510925    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"]
Oct 30 01:22:32.511845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.511827    5104 cpu_manager.go:414] "containerMap: removing container" podUID="989267ea-10a8-4299-a24e-ee194aa3616e" containerName="revoke-keys"
Oct 30 01:22:32.511914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.511845    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989267ea-10a8-4299-a24e-ee194aa3616e" containerName="revoke-keys"
Oct 30 01:22:32.512038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.512028    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="989267ea-10a8-4299-a24e-ee194aa3616e" containerName="revoke-keys"
Oct 30 01:22:32.516289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.516265    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.533286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.533241    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:22:32.535786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.533384    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:22:32.539794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.539762    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"]
Oct 30 01:22:32.541432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.541400    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:32.541432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.541405    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:22:32.651515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.651208    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.651515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.651386    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.651841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.651574    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82bc\" (UniqueName: \"kubernetes.io/projected/4689b723-1fce-46a9-b587-3ed813fcde23-kube-api-access-x82bc\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.753274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.753187    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.753436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.753288    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.753436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.753338    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82bc\" (UniqueName: \"kubernetes.io/projected/4689b723-1fce-46a9-b587-3ed813fcde23-kube-api-access-x82bc\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.754105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.754074    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.755880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.755848    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.773797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.773764    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82bc\" (UniqueName: \"kubernetes.io/projected/4689b723-1fce-46a9-b587-3ed813fcde23-kube-api-access-x82bc\") pod \"maas-api-revoke-sub-9b93c072-f5hv8\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:32.835081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:32.835047    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:33.121334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:33.121217    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"]
Oct 30 01:22:33.134815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:33.134765    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb342b8-vdmvx"]
Oct 30 01:22:33.175668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:33.175631    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"]
Oct 30 01:22:33.432394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:33.432354    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8" event={"ID":"4689b723-1fce-46a9-b587-3ed813fcde23","Type":"ContainerStarted","Data":"9bb4ee567a32b9350f67d8c842bdefbeccb38525d3773aced9b6e3dd9b3fed5d"}
Oct 30 01:22:34.446969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.446930    5104 generic.go:356] "Generic (PLEG): container finished" podID="4689b723-1fce-46a9-b587-3ed813fcde23" containerID="5518980b575ee7cf153aeab75138fa4d3cb1545e201180cc3d0b6d45b67aee94" exitCode=0
Oct 30 01:22:34.447398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.447034    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8" event={"ID":"4689b723-1fce-46a9-b587-3ed813fcde23","Type":"ContainerDied","Data":"5518980b575ee7cf153aeab75138fa4d3cb1545e201180cc3d0b6d45b67aee94"}
Oct 30 01:22:34.762884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.762844    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cab7767-f495-41db-91c7-2542fb0af7cf" path="/var/lib/kubelet/pods/8cab7767-f495-41db-91c7-2542fb0af7cf/volumes"
Oct 30 01:22:34.885688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.885644    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"]
Oct 30 01:22:34.899002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.898967    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:34.926547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.913817    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d32079\""
Oct 30 01:22:34.926547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.914705    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d32079-dockercfg-7z8kk\""
Oct 30 01:22:34.983790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.963173    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:34.983790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:34.963275    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7vj\" (UniqueName: \"kubernetes.io/projected/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-kube-api-access-bq7vj\") pod \"payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:35.004702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.004506    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"]
Oct 30 01:22:35.183480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.172830    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:35.332861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.332822    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7vj\" (UniqueName: \"kubernetes.io/projected/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-kube-api-access-bq7vj\") pod \"payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:35.344917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.187862    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:35.382235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.382198    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7vj\" (UniqueName: \"kubernetes.io/projected/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-kube-api-access-bq7vj\") pod \"payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:35.401967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.401927    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:35.788131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.788094    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:35.789381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.789337    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"]
Oct 30 01:22:35.821018 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:35.820972    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45040b2d_a8b3_4d6e_95a0_f450aeae6ec9.slice/crio-08edb3ab2840ba77e3bcc011722cdbb7926c59bdebaeb1ab248d12e86b29d230 WatchSource:0}: Error finding container 08edb3ab2840ba77e3bcc011722cdbb7926c59bdebaeb1ab248d12e86b29d230: Status 404 returned error can't find the container with id 08edb3ab2840ba77e3bcc011722cdbb7926c59bdebaeb1ab248d12e86b29d230
Oct 30 01:22:35.823841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.823817    5104 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:22:35.874239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.872435    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-xks-ca\") pod \"4689b723-1fce-46a9-b587-3ed813fcde23\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") "
Oct 30 01:22:35.874239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.872574    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-openshift-ca\") pod \"4689b723-1fce-46a9-b587-3ed813fcde23\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") "
Oct 30 01:22:35.874239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.872713    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4689b723-1fce-46a9-b587-3ed813fcde23-kube-api-access-x82bc\" (UniqueName: \"kubernetes.io/projected/4689b723-1fce-46a9-b587-3ed813fcde23-kube-api-access-x82bc\") pod \"4689b723-1fce-46a9-b587-3ed813fcde23\" (UID: \"4689b723-1fce-46a9-b587-3ed813fcde23\") "
Oct 30 01:22:35.874239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.873167    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-openshift-ca" pod "4689b723-1fce-46a9-b587-3ed813fcde23" (UID: "4689b723-1fce-46a9-b587-3ed813fcde23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:35.874239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.873713    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.878915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.878885    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4689b723-1fce-46a9-b587-3ed813fcde23-kube-api-access-x82bc" pod "4689b723-1fce-46a9-b587-3ed813fcde23" (UID: "4689b723-1fce-46a9-b587-3ed813fcde23"). InnerVolumeSpecName "kube-api-access-x82bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:35.882716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.882652    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-xks-ca" pod "4689b723-1fce-46a9-b587-3ed813fcde23" (UID: "4689b723-1fce-46a9-b587-3ed813fcde23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:35.975419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.975325    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x82bc\" (UniqueName: \"kubernetes.io/projected/4689b723-1fce-46a9-b587-3ed813fcde23-kube-api-access-x82bc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.975419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:35.975365    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4689b723-1fce-46a9-b587-3ed813fcde23-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.486510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:36.484258    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw" event={"ID":"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9","Type":"ContainerStarted","Data":"08edb3ab2840ba77e3bcc011722cdbb7926c59bdebaeb1ab248d12e86b29d230"}
Oct 30 01:22:36.490487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:36.488000    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8" event={"ID":"4689b723-1fce-46a9-b587-3ed813fcde23","Type":"ContainerDied","Data":"9bb4ee567a32b9350f67d8c842bdefbeccb38525d3773aced9b6e3dd9b3fed5d"}
Oct 30 01:22:36.490487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:36.488059    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb4ee567a32b9350f67d8c842bdefbeccb38525d3773aced9b6e3dd9b3fed5d"
Oct 30 01:22:36.490487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:36.488122    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"
Oct 30 01:22:38.412479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.412170    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"]
Oct 30 01:22:38.413825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.413641    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4689b723-1fce-46a9-b587-3ed813fcde23" containerName="revoke-keys"
Oct 30 01:22:38.413825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.413671    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4689b723-1fce-46a9-b587-3ed813fcde23" containerName="revoke-keys"
Oct 30 01:22:38.414050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.413874    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4689b723-1fce-46a9-b587-3ed813fcde23" containerName="revoke-keys"
Oct 30 01:22:38.421588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.421556    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.437726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.437679    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"]
Oct 30 01:22:38.439612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.439585    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:38.439767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.439691    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:22:38.453575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.450609    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:22:38.453575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.450832    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:22:38.516483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.506711    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e298f917-fac1-454e-92a1-15647183312c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.516483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.506783    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e298f917-fac1-454e-92a1-15647183312c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.516483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.507191    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/e298f917-fac1-454e-92a1-15647183312c-kube-api-access-6zr6r\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.610490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.609508    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/e298f917-fac1-454e-92a1-15647183312c-kube-api-access-6zr6r\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.610490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.609629    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e298f917-fac1-454e-92a1-15647183312c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.610490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.609667    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e298f917-fac1-454e-92a1-15647183312c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.611100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.611077    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e298f917-fac1-454e-92a1-15647183312c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.624998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.624930    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e298f917-fac1-454e-92a1-15647183312c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.634970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.634939    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/e298f917-fac1-454e-92a1-15647183312c-kube-api-access-6zr6r\") pod \"maas-api-revoke-sub-85b1c5e7-2r7w9\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:38.759432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.759407    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:22:38.766407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:38.766375    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:39.344940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.327368    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"]
Oct 30 01:22:39.357543 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:22:39.357510    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:22:39.458494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.457927    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:22:39.459789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.459772    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:22:39.466796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.466767    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:22:39.469652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.469622    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:22:39.526617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.526589    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:22:39.528164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.526639    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:39.551200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.551166    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw" event={"ID":"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9","Type":"ContainerStarted","Data":"5df04ae08139cc1207d9c2c2959ca0c69e6fb4894313f1f96729c24e368e1317"}
Oct 30 01:22:39.551442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.551415    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:39.553242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.553140    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9" event={"ID":"e298f917-fac1-454e-92a1-15647183312c","Type":"ContainerStarted","Data":"4ab41822e9662db31779f36d3cdfcee719bb1275f43d66871d82523afaca16c1"}
Oct 30 01:22:39.605575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:39.602556    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw" podStartSLOduration=2.770679509 podStartE2EDuration="5.602532334s" totalImagesPullingTime="2.831852825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:39.599971703 +0000 UTC m=+2401.147662202" watchObservedRunningTime="2026-10-30 01:22:39.602532334 +0000 UTC m=+2401.150222833"
Oct 30 01:22:40.569145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.569108    5104 generic.go:356] "Generic (PLEG): container finished" podID="70785039-29f0-4fee-ae6e-3719b53420c0" containerID="a70cc2a52a6d25ab3952f306807882193d04bac7026d931c0709e4c5531d8bec" exitCode=0
Oct 30 01:22:40.570201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.569324    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" event={"ID":"70785039-29f0-4fee-ae6e-3719b53420c0","Type":"ContainerDied","Data":"a70cc2a52a6d25ab3952f306807882193d04bac7026d931c0709e4c5531d8bec"}
Oct 30 01:22:40.570201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.569876    5104 scope.go:122] "RemoveContainer" containerID="a70cc2a52a6d25ab3952f306807882193d04bac7026d931c0709e4c5531d8bec"
Oct 30 01:22:40.572321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.572284    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:40.586610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.572364    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:40.586610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.572399    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:40.627836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.627792    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9" event={"ID":"e298f917-fac1-454e-92a1-15647183312c","Type":"ContainerStarted","Data":"a6aa2a2b46ae40f12062e4c01f0585e2c1acfce3ff60690951dff6f46bd2b8a4"}
Oct 30 01:22:40.728282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:40.728217    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9" podStartSLOduration=2.728196443 podStartE2EDuration="2.728196443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:40.692780426 +0000 UTC m=+2402.240470923" watchObservedRunningTime="2026-10-30 01:22:40.728196443 +0000 UTC m=+2402.275886941"
Oct 30 01:22:41.628191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:41.628146    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" event={"ID":"70785039-29f0-4fee-ae6e-3719b53420c0","Type":"ContainerStarted","Data":"501772db4abba02a697620530d64a7ddfe24df6290c45aa63b43581aa16e4e4e"}
Oct 30 01:22:41.630901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:41.630868    5104 generic.go:356] "Generic (PLEG): container finished" podID="e298f917-fac1-454e-92a1-15647183312c" containerID="a6aa2a2b46ae40f12062e4c01f0585e2c1acfce3ff60690951dff6f46bd2b8a4" exitCode=0
Oct 30 01:22:41.631138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:41.630936    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9" event={"ID":"e298f917-fac1-454e-92a1-15647183312c","Type":"ContainerDied","Data":"a6aa2a2b46ae40f12062e4c01f0585e2c1acfce3ff60690951dff6f46bd2b8a4"}
Oct 30 01:22:41.927205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:41.927177    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:22:41.927449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:41.927203    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body=
Oct 30 01:22:41.927621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:41.927602    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused"
Oct 30 01:22:42.863231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:42.860999    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"]
Oct 30 01:22:42.944401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:42.927731    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:42.944401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:42.929357    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body=
Oct 30 01:22:42.944401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:42.929414    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused"
Oct 30 01:22:42.979480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:42.967555    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"]
Oct 30 01:22:43.068421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.068324    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ksv\" (UniqueName: \"kubernetes.io/projected/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-kube-api-access-g8ksv\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.068946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.068891    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.069099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.069069    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.171104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.171064    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.171261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.171169    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.171346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.171330    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ksv\" (UniqueName: \"kubernetes.io/projected/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-kube-api-access-g8ksv\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.172030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.172003    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.248507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.245061    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.255478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.249576    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ksv\" (UniqueName: \"kubernetes.io/projected/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-kube-api-access-g8ksv\") pod \"maas-api-revoke-sub-f6ef162b-9clfr\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.357253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.357220    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:43.398532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.382531    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e298f917-fac1-454e-92a1-15647183312c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e298f917-fac1-454e-92a1-15647183312c-maas-api-xks-ca\") pod \"e298f917-fac1-454e-92a1-15647183312c\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") "
Oct 30 01:22:43.398532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.382604    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e298f917-fac1-454e-92a1-15647183312c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e298f917-fac1-454e-92a1-15647183312c-maas-api-openshift-ca\") pod \"e298f917-fac1-454e-92a1-15647183312c\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") "
Oct 30 01:22:43.398532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.382992    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e298f917-fac1-454e-92a1-15647183312c-kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/e298f917-fac1-454e-92a1-15647183312c-kube-api-access-6zr6r\") pod \"e298f917-fac1-454e-92a1-15647183312c\" (UID: \"e298f917-fac1-454e-92a1-15647183312c\") "
Oct 30 01:22:43.398532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.384757    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e298f917-fac1-454e-92a1-15647183312c-maas-api-openshift-ca" pod "e298f917-fac1-454e-92a1-15647183312c" (UID: "e298f917-fac1-454e-92a1-15647183312c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:43.439231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.429599    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e298f917-fac1-454e-92a1-15647183312c-maas-api-xks-ca" pod "e298f917-fac1-454e-92a1-15647183312c" (UID: "e298f917-fac1-454e-92a1-15647183312c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:43.441689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.441649    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e298f917-fac1-454e-92a1-15647183312c-kube-api-access-6zr6r" pod "e298f917-fac1-454e-92a1-15647183312c" (UID: "e298f917-fac1-454e-92a1-15647183312c"). InnerVolumeSpecName "kube-api-access-6zr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:43.466813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.466782    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:43.505480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.499122    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/e298f917-fac1-454e-92a1-15647183312c-kube-api-access-6zr6r\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.505480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.499167    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e298f917-fac1-454e-92a1-15647183312c-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.505480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.499185    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e298f917-fac1-454e-92a1-15647183312c-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.696482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.694189    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9" event={"ID":"e298f917-fac1-454e-92a1-15647183312c","Type":"ContainerDied","Data":"4ab41822e9662db31779f36d3cdfcee719bb1275f43d66871d82523afaca16c1"}
Oct 30 01:22:43.696482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.694258    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ab41822e9662db31779f36d3cdfcee719bb1275f43d66871d82523afaca16c1"
Oct 30 01:22:43.696482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.694325    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"
Oct 30 01:22:43.819662 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:43.819624    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5059a4_78ed_414e_93b1_37fd49b2bcfc.slice/crio-8139b37e836c0eb14cbd6a24e7d1a00dde52db0683a0773cf46b60779bde5b6a WatchSource:0}: Error finding container 8139b37e836c0eb14cbd6a24e7d1a00dde52db0683a0773cf46b60779bde5b6a: Status 404 returned error can't find the container with id 8139b37e836c0eb14cbd6a24e7d1a00dde52db0683a0773cf46b60779bde5b6a
Oct 30 01:22:43.821518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.821487    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"]
Oct 30 01:22:43.932735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.932683    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body=
Oct 30 01:22:43.933288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:43.932761    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused"
Oct 30 01:22:44.729625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:44.729573    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr" event={"ID":"bb5059a4-78ed-414e-93b1-37fd49b2bcfc","Type":"ContainerStarted","Data":"5f9120c2800b7fd6bd4de4edfd22780a74e95450657fb323a76c1f29fe2ad990"}
Oct 30 01:22:44.729845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:44.729650    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr" event={"ID":"bb5059a4-78ed-414e-93b1-37fd49b2bcfc","Type":"ContainerStarted","Data":"8139b37e836c0eb14cbd6a24e7d1a00dde52db0683a0773cf46b60779bde5b6a"}
Oct 30 01:22:44.813886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:44.811315    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr" podStartSLOduration=2.811293991 podStartE2EDuration="2.811293991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:44.807724565 +0000 UTC m=+2406.355415061" watchObservedRunningTime="2026-10-30 01:22:44.811293991 +0000 UTC m=+2406.358984494"
Oct 30 01:22:44.934782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:44.934717    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body=
Oct 30 01:22:44.935307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:44.934806    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused"
Oct 30 01:22:45.762552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:45.761792    5104 generic.go:356] "Generic (PLEG): container finished" podID="bb5059a4-78ed-414e-93b1-37fd49b2bcfc" containerID="5f9120c2800b7fd6bd4de4edfd22780a74e95450657fb323a76c1f29fe2ad990" exitCode=0
Oct 30 01:22:45.762552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:45.762055    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr" event={"ID":"bb5059a4-78ed-414e-93b1-37fd49b2bcfc","Type":"ContainerDied","Data":"5f9120c2800b7fd6bd4de4edfd22780a74e95450657fb323a76c1f29fe2ad990"}
Oct 30 01:22:45.990399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:45.987652    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body=
Oct 30 01:22:45.990399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:45.987727    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused"
Oct 30 01:22:46.934819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:46.934276    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body=
Oct 30 01:22:46.934819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:46.934373    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused"
Oct 30 01:22:47.111532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.110707    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:22:47.111532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.110831    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:47.416489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.414073    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:47.520860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.520305    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-kube-api-access-g8ksv\" (UniqueName: \"kubernetes.io/projected/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-kube-api-access-g8ksv\") pod \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") "
Oct 30 01:22:47.521071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.521006    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-xks-ca\") pod \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") "
Oct 30 01:22:47.521567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.521536    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-openshift-ca\") pod \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\" (UID: \"bb5059a4-78ed-414e-93b1-37fd49b2bcfc\") "
Oct 30 01:22:47.522903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.522870    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-openshift-ca" pod "bb5059a4-78ed-414e-93b1-37fd49b2bcfc" (UID: "bb5059a4-78ed-414e-93b1-37fd49b2bcfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:47.523813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.523753    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-xks-ca" pod "bb5059a4-78ed-414e-93b1-37fd49b2bcfc" (UID: "bb5059a4-78ed-414e-93b1-37fd49b2bcfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:47.534834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.532994    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-kube-api-access-g8ksv" pod "bb5059a4-78ed-414e-93b1-37fd49b2bcfc" (UID: "bb5059a4-78ed-414e-93b1-37fd49b2bcfc"). InnerVolumeSpecName "kube-api-access-g8ksv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:47.534834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.533324    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"]
Oct 30 01:22:47.563724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.563689    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"]
Oct 30 01:22:47.565378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.565349    5104 cpu_manager.go:414] "containerMap: removing container" podUID="bb5059a4-78ed-414e-93b1-37fd49b2bcfc" containerName="revoke-keys"
Oct 30 01:22:47.565771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.565751    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5059a4-78ed-414e-93b1-37fd49b2bcfc" containerName="revoke-keys"
Oct 30 01:22:47.566779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.566760    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e298f917-fac1-454e-92a1-15647183312c" containerName="revoke-keys"
Oct 30 01:22:47.566914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.566901    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e298f917-fac1-454e-92a1-15647183312c" containerName="revoke-keys"
Oct 30 01:22:47.567286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.567271    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5059a4-78ed-414e-93b1-37fd49b2bcfc" containerName="revoke-keys"
Oct 30 01:22:47.567416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.567405    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e298f917-fac1-454e-92a1-15647183312c" containerName="revoke-keys"
Oct 30 01:22:47.635910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.635872    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8ksv\" (UniqueName: \"kubernetes.io/projected/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-kube-api-access-g8ksv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:47.635910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.635911    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:47.636190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.635927    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5059a4-78ed-414e-93b1-37fd49b2bcfc-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:47.762259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.761568    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:47.762259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.761816    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"]
Oct 30 01:22:47.879560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.822970    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-add1bb-openshift-default-dockercfg-hsjlp\""
Oct 30 01:22:47.944163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.941889    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body=
Oct 30 01:22:47.944163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.941954    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused"
Oct 30 01:22:47.970107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.969984    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"
Oct 30 01:22:47.970107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.969980    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr" event={"ID":"bb5059a4-78ed-414e-93b1-37fd49b2bcfc","Type":"ContainerDied","Data":"8139b37e836c0eb14cbd6a24e7d1a00dde52db0683a0773cf46b60779bde5b6a"}
Oct 30 01:22:47.970107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:47.970061    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8139b37e836c0eb14cbd6a24e7d1a00dde52db0683a0773cf46b60779bde5b6a"
Oct 30 01:22:48.019985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.019885    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.019985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.019943    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hd9\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-kube-api-access-t2hd9\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.020237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.019987    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.020237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.020043    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.020237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.020094    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.020237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.020172    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ca365-14ab-4b2f-a933-60387a9c687b-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.020501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.020272    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.020501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.020302    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.020501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.020427    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.122597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.122558    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.123212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.123187    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.123430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.123413    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ca365-14ab-4b2f-a933-60387a9c687b-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.123640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.123625    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.123760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.123746    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.123958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.123946    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.124201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.124185    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.124312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.124298    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hd9\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-kube-api-access-t2hd9\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.124428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.124415    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.125475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.125433    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.125660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.125636    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.125861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.125837    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.126103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.126085    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.126850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.126803    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ca365-14ab-4b2f-a933-60387a9c687b-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.127777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.127649    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.131854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.131189    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.149109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.149073    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.149277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.149068    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hd9\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-kube-api-access-t2hd9\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.276537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.271333    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:48.305475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.284943    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"]
Oct 30 01:22:48.305475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.295892    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.329843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.329806    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7f0263-openshift-default-dockercfg-n5lgt\""
Oct 30 01:22:48.330788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.330747    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"]
Oct 30 01:22:48.331863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.331823    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-certs\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.331877    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b131465-6995-4c54-909b-49205c47f220-istiod-ca-cert\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.331906    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kc5c\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-kube-api-access-7kc5c\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.332056    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-istio-token\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.332096    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-credential-socket\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.332236    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b131465-6995-4c54-909b-49205c47f220-istio-podinfo\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.332273    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-envoy\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.332350    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-socket\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.332430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.332385    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-data\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.433835    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b131465-6995-4c54-909b-49205c47f220-istio-podinfo\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.433894    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-envoy\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.433964    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-socket\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.433995    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-data\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.435078    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-certs\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.435124    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b131465-6995-4c54-909b-49205c47f220-istiod-ca-cert\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.435150    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kc5c\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-kube-api-access-7kc5c\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.435303    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-istio-token\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.435345    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-credential-socket\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.435801    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-credential-socket\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.436056    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-socket\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.436298    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-data\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.436570    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-certs\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.453137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.437193    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b131465-6995-4c54-909b-49205c47f220-istiod-ca-cert\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.488532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.466996    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-istio-token\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.497049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.496922    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-envoy\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.502478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.502229    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b131465-6995-4c54-909b-49205c47f220-istio-podinfo\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.756833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.620533    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kc5c\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-kube-api-access-7kc5c\") pod \"e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") " pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:48.759848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:48.759812    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:49.173689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.173636    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"]
Oct 30 01:22:49.184262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.181628    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:49.184262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.181735    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:49.184262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.181781    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:49.464364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.460000    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"]
Oct 30 01:22:49.536435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.530310    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:22:49.536435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.531530    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:22:49.585480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.582515    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:22:49.633136 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:49.633101    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b131465_6995_4c54_909b_49205c47f220.slice/crio-c5dccab87174a375efd557d19e7fe8daeafb71a3f74cbb31a93580e3bef65f2d WatchSource:0}: Error finding container c5dccab87174a375efd557d19e7fe8daeafb71a3f74cbb31a93580e3bef65f2d: Status 404 returned error can't find the container with id c5dccab87174a375efd557d19e7fe8daeafb71a3f74cbb31a93580e3bef65f2d
Oct 30 01:22:49.637229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.637183    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:49.637369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.637283    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:49.637369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:49.637326    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:50.070634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.070543    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" event={"ID":"8e1ca365-14ab-4b2f-a933-60387a9c687b","Type":"ContainerStarted","Data":"6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3"}
Oct 30 01:22:50.070634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.070610    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" event={"ID":"8e1ca365-14ab-4b2f-a933-60387a9c687b","Type":"ContainerStarted","Data":"9eff245986fe750450271cbbc85b897a84039f945989ad137a0bde4e4d34904c"}
Oct 30 01:22:50.083881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.083834    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" event={"ID":"1b131465-6995-4c54-909b-49205c47f220","Type":"ContainerStarted","Data":"ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1"}
Oct 30 01:22:50.084082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.083901    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" event={"ID":"1b131465-6995-4c54-909b-49205c47f220","Type":"ContainerStarted","Data":"c5dccab87174a375efd557d19e7fe8daeafb71a3f74cbb31a93580e3bef65f2d"}
Oct 30 01:22:50.135717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.135640    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podStartSLOduration=3.13561707 podStartE2EDuration="3.13561707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:50.116897309 +0000 UTC m=+2411.664587806" watchObservedRunningTime="2026-10-30 01:22:50.13561707 +0000 UTC m=+2411.683307567"
Oct 30 01:22:50.163067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.163006    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podStartSLOduration=2.162990735 podStartE2EDuration="2.162990735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:50.158515372 +0000 UTC m=+2411.706205868" watchObservedRunningTime="2026-10-30 01:22:50.162990735 +0000 UTC m=+2411.710681231"
Oct 30 01:22:50.275289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.275255    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:50.276050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.276023    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 01:22:50.276241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.276214    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 01:22:50.615650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.615614    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:22:50.834496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.824824    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:50.843925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:50.843886    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:51.206476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.192432    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:51.206476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.203543    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:22:51.285515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.272417    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 01:22:51.285515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.272519    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 01:22:51.568635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.568544    5104 scope.go:122] "RemoveContainer" containerID="522df8d322cfc465e7b1f6f1d0346ac9572e8e00c848648de86b582bda8dfc98"
Oct 30 01:22:51.616380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.616353    5104 scope.go:122] "RemoveContainer" containerID="36aaa391b42240784b85ff80f26b306b6d9f7a06ea1757be8ed2e187bbd20dcb"
Oct 30 01:22:51.631310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.631147    5104 scope.go:122] "RemoveContainer" containerID="003c2092bd8b625fe16e0361a7d83299bce2036960d8b0e8e3bcfe4273dce483"
Oct 30 01:22:51.646607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.646305    5104 scope.go:122] "RemoveContainer" containerID="a978b0d2be51c5c1d40b966be13a12219e6b1879e218848b7f07f5f6d0d20d6e"
Oct 30 01:22:51.660505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.660475    5104 scope.go:122] "RemoveContainer" containerID="26d7d84623e5fab9b1f420f392ea770d49ab5b02767ec18b8bbfd0918a1face2"
Oct 30 01:22:51.672355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.672334    5104 scope.go:122] "RemoveContainer" containerID="7f9987a9a9a933a5956418aadcff2d881e5abf07b66ffcb2852c86d11422b771"
Oct 30 01:22:51.726430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:51.726395    5104 scope.go:122] "RemoveContainer" containerID="65108e5c1cf269bc8dd3dc5110799b6ca27cb9c353da97dc6cd5e3ed59a64e19"
Oct 30 01:22:52.295480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:52.290785    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 01:22:52.295480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:52.290851    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 01:22:52.534334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:52.534286    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk" podUID="83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" containerName="payload-processing" containerID="cri-o://4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f" gracePeriod=25
Oct 30 01:22:53.139483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.102410    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:22:53.247306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.247207    5104 generic.go:356] "Generic (PLEG): container finished" podID="83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" containerID="4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f" exitCode=0
Oct 30 01:22:53.248661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.248601    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"
Oct 30 01:22:53.248973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.248945    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk" event={"ID":"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353","Type":"ContainerDied","Data":"4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f"}
Oct 30 01:22:53.249083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.248996    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk" event={"ID":"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353","Type":"ContainerDied","Data":"8654838a97137778f2ff3292ddbd75a2a8febdacf3b5cd41eb8eceb1d3e65394"}
Oct 30 01:22:53.249083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.249018    5104 scope.go:122] "RemoveContainer" containerID="4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f"
Oct 30 01:22:53.269603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.269577    5104 scope.go:122] "RemoveContainer" containerID="4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f"
Oct 30 01:22:53.269879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.269855    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-plugins-config-volume\") pod \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") "
Oct 30 01:22:53.270263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.270240    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-kube-api-access-db87m\" (UniqueName: \"kubernetes.io/projected/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-kube-api-access-db87m\") pod \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\" (UID: \"83f0591a-4b9d-4c4c-9b4d-0c4b958a8353\") "
Oct 30 01:22:53.270263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.270251    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-plugins-config-volume" pod "83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" (UID: "83f0591a-4b9d-4c4c-9b4d-0c4b958a8353"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:53.271031 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:22:53.271003    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f\": container with ID starting with 4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f not found: ID does not exist" containerID="4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f"
Oct 30 01:22:53.271143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.271038    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f"} err="failed to get container status \"4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f\": rpc error: code = NotFound desc = could not find container \"4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f\": container with ID starting with 4e2a1358b31f6d0d25a12e8d663f749299d0073f7bd778804f75dbf93c12ee7f not found: ID does not exist"
Oct 30 01:22:53.271402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.271378    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:53.308478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.291322    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 01:22:53.308478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.291390    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 01:22:53.312224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.311577    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-kube-api-access-db87m" pod "83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" (UID: "83f0591a-4b9d-4c4c-9b4d-0c4b958a8353"). InnerVolumeSpecName "kube-api-access-db87m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:53.400481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.387654    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db87m\" (UniqueName: \"kubernetes.io/projected/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353-kube-api-access-db87m\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:22:53.619747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.618615    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"]
Oct 30 01:22:53.629416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:53.629376    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-6d67f7db44-ll6wk"]
Oct 30 01:22:54.260297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.260263    5104 generic.go:356] "Generic (PLEG): container finished" podID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerID="9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d" exitCode=0
Oct 30 01:22:54.260546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.260332    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" event={"ID":"f8b35fbd-906b-450d-99a0-b5b0f405b607","Type":"ContainerDied","Data":"9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d"}
Oct 30 01:22:54.260951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.260930    5104 scope.go:122] "RemoveContainer" containerID="9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d"
Oct 30 01:22:54.263106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.263073    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:54.263228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.263163    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:54.263228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.263215    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:22:54.272056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.272032    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 01:22:54.272145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.272079    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 01:22:54.626888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.626790    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" path="/var/lib/kubelet/pods/83f0591a-4b9d-4c4c-9b4d-0c4b958a8353/volumes"
Oct 30 01:22:54.627416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.627388    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"]
Oct 30 01:22:54.628395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.628369    5104 cpu_manager.go:414] "containerMap: removing container" podUID="83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" containerName="payload-processing"
Oct 30 01:22:54.628395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.628393    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" containerName="payload-processing"
Oct 30 01:22:54.628758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.628742    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f0591a-4b9d-4c4c-9b4d-0c4b958a8353" containerName="payload-processing"
Oct 30 01:22:54.633262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.633239    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:54.650098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.650067    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7f0263-dockercfg-jnxfr\""
Oct 30 01:22:54.650350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.650337    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7f0263\""
Oct 30 01:22:54.655536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.652989    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"]
Oct 30 01:22:54.757371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.746416    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s84m\" (UniqueName: \"kubernetes.io/projected/a1fc686c-34eb-442b-80f1-623d1f780820-kube-api-access-9s84m\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:54.757371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.746680    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fc686c-34eb-442b-80f1-623d1f780820-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:54.906537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.897267    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s84m\" (UniqueName: \"kubernetes.io/projected/a1fc686c-34eb-442b-80f1-623d1f780820-kube-api-access-9s84m\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:54.906537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.897439    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fc686c-34eb-442b-80f1-623d1f780820-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:54.906537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.898658    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fc686c-34eb-442b-80f1-623d1f780820-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:54.967253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:54.967214    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s84m\" (UniqueName: \"kubernetes.io/projected/a1fc686c-34eb-442b-80f1-623d1f780820-kube-api-access-9s84m\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:55.030607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.030571    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"]
Oct 30 01:22:55.200544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.188770    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.278349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.239189    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.278349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.239299    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4cmd\" (UniqueName: \"kubernetes.io/projected/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-kube-api-access-w4cmd\") pod \"payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.307480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.306553    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:55.426514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.416558    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"]
Oct 30 01:22:55.426514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.417780    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4cmd\" (UniqueName: \"kubernetes.io/projected/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-kube-api-access-w4cmd\") pod \"payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.426514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.418215    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.470737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.470697    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 01:22:55.472896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.472843    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 01:22:55.485513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.485194    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.564482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.561031    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4cmd\" (UniqueName: \"kubernetes.io/projected/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-kube-api-access-w4cmd\") pod \"payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.631525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.631432    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:55.731540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:55.730711    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" event={"ID":"f8b35fbd-906b-450d-99a0-b5b0f405b607","Type":"ContainerStarted","Data":"49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8"}
Oct 30 01:22:56.030580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:56.029473    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:22:56.113801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:56.113758    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 30 01:22:56.156475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:56.146744    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 30 01:22:56.335588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:56.335432    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body=
Oct 30 01:22:56.336011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:56.335973    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused"
Oct 30 01:22:56.426488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:56.397970    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"]
Oct 30 01:22:56.867589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:56.867494    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z" event={"ID":"a1fc686c-34eb-442b-80f1-623d1f780820","Type":"ContainerStarted","Data":"98ea7869fca3daedf4dab42f0a7d7040eea1f7d8f108c8cac05b98e63b7cbe84"}
Oct 30 01:22:57.045144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.045103    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 30 01:22:57.062570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.062522    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 30 01:22:57.224522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.223761    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"]
Oct 30 01:22:57.227990 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:22:57.227934    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164a6d4e_d4c1_4a92_895a_2a0bf6a969e7.slice/crio-0db8ae7b559fdb3ea7b08fa8e38227cfc2c7a86aceb4c5258c96cd743b4642a5 WatchSource:0}: Error finding container 0db8ae7b559fdb3ea7b08fa8e38227cfc2c7a86aceb4c5258c96cd743b4642a5: Status 404 returned error can't find the container with id 0db8ae7b559fdb3ea7b08fa8e38227cfc2c7a86aceb4c5258c96cd743b4642a5
Oct 30 01:22:57.763065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.763019    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:57.764502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.764242    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:57.765707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.765675    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:22:57.943499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.941774    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5" event={"ID":"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7","Type":"ContainerStarted","Data":"fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896"}
Oct 30 01:22:57.943499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.941840    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5" event={"ID":"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7","Type":"ContainerStarted","Data":"0db8ae7b559fdb3ea7b08fa8e38227cfc2c7a86aceb4c5258c96cd743b4642a5"}
Oct 30 01:22:57.943499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.941911    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:22:57.947127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.946987    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z" event={"ID":"a1fc686c-34eb-442b-80f1-623d1f780820","Type":"ContainerStarted","Data":"1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734"}
Oct 30 01:22:57.947127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:57.947049    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:22:58.008138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:58.008100    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 30 01:22:58.027643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:58.027536    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 30 01:22:58.033585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:58.033450    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5" podStartSLOduration=4.033427262 podStartE2EDuration="4.033427262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:58.007473918 +0000 UTC m=+2419.555164416" watchObservedRunningTime="2026-10-30 01:22:58.033427262 +0000 UTC m=+2419.581117759"
Oct 30 01:22:58.051138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:58.051086    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z" podStartSLOduration=4.05106744 podStartE2EDuration="4.05106744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:58.049951633 +0000 UTC m=+2419.597642130" watchObservedRunningTime="2026-10-30 01:22:58.05106744 +0000 UTC m=+2419.598757936"
Oct 30 01:22:59.012639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:59.012603    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 30 01:22:59.013184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:22:59.012664    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 30 01:23:00.004159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.004117    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 30 01:23:00.004543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.004506    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 30 01:23:00.037345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.037310    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"]
Oct 30 01:23:00.042831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.042800    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"]
Oct 30 01:23:00.048645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.048612    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab03cb2a-pn25v"]
Oct 30 01:23:00.054938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.054907    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdd3d36-vgc4v"]
Oct 30 01:23:00.606656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.606606    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4283b9-51c1-4278-b16d-ad517abf10a9" path="/var/lib/kubelet/pods/0a4283b9-51c1-4278-b16d-ad517abf10a9/volumes"
Oct 30 01:23:00.607328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:00.607307    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892961d2-a1b7-4a4d-9a82-47faf0d31c4e" path="/var/lib/kubelet/pods/892961d2-a1b7-4a4d-9a82-47faf0d31c4e/volumes"
Oct 30 01:23:01.004988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:01.004948    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 30 01:23:01.005192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:01.005030    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 30 01:23:01.013102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:01.013078    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:23:02.170729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:02.170693    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:23:02.172596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:02.172565    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:23:06.484350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:06.484308    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"]
Oct 30 01:23:08.159893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:08.159847    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"]
Oct 30 01:23:09.005018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:09.004988    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:23:09.005216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:09.005058    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:23:09.130932 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:09.130887    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:23:15.750180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.750131    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"]
Oct 30 01:23:15.757383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.757333    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.817200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.817163    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"]
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.819781    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-73831d-openshift-default-dockercfg-2pzkm\""
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.821699    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istiod-ca-cert\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.821757    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-podinfo\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.821840    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-credential-socket\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.822017    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-token\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.822055    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wtb\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-kube-api-access-j9wtb\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.822151    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-data\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.822285    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-certs\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.822359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-envoy\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.822862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.822502    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-socket\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925262    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-certs\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925413    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-envoy\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925557    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-socket\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925619    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istiod-ca-cert\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925736    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-podinfo\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925783    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-certs\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925849    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-credential-socket\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.925962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925948    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-token\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.926023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925978    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wtb\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-kube-api-access-j9wtb\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.926023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.925984    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-socket\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.926132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.926079    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-data\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.926230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.926179    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-credential-socket\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.926405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.926373    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istiod-ca-cert\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.926405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.926399    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-data\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.928363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.928336    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-envoy\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.928636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.928617    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-podinfo\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.951295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.951221    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-token\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:15.956470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:15.956367    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wtb\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-kube-api-access-j9wtb\") pod \"e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") " pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:16.136531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:16.130817    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:16.779844 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:23:16.779808    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93ddf6d_66f1_470c_aa41_91ca7a4c9996.slice/crio-6a224fb4bec90c4be721edd2663b183ee794c50adff237900253eb734473ae95 WatchSource:0}: Error finding container 6a224fb4bec90c4be721edd2663b183ee794c50adff237900253eb734473ae95: Status 404 returned error can't find the container with id 6a224fb4bec90c4be721edd2663b183ee794c50adff237900253eb734473ae95
Oct 30 01:23:16.780293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:16.780070    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"]
Oct 30 01:23:16.783806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:16.783761    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:16.783975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:16.783856    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:16.783975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:16.783898    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:17.441313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:17.441270    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" event={"ID":"a93ddf6d-66f1-470c-aa41-91ca7a4c9996","Type":"ContainerStarted","Data":"acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3"}
Oct 30 01:23:17.441551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:17.441343    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" event={"ID":"a93ddf6d-66f1-470c-aa41-91ca7a4c9996","Type":"ContainerStarted","Data":"6a224fb4bec90c4be721edd2663b183ee794c50adff237900253eb734473ae95"}
Oct 30 01:23:17.495923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:17.495854    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podStartSLOduration=2.495831458 podStartE2EDuration="2.495831458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:17.49292595 +0000 UTC m=+2439.040616447" watchObservedRunningTime="2026-10-30 01:23:17.495831458 +0000 UTC m=+2439.043521954"
Oct 30 01:23:18.133267 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:18.133222    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:18.139328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:18.139300    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:18.453879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:18.453845    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:18.454904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:18.454880    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:23:20.014565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:20.014519    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"]
Oct 30 01:23:21.032717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.032674    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"]
Oct 30 01:23:21.040711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.040678    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.062495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.062435    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"]
Oct 30 01:23:21.070325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.070290    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d3ed60-openshift-default-dockercfg-w4n99\""
Oct 30 01:23:21.106474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.106412    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d294-abf2-4b68-bec1-f29971fca466-istiod-ca-cert\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.106793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.106751    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-credential-socket\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.106958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.106940    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwzv\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-kube-api-access-xzwzv\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.107117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.107098    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-envoy\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.107357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.107336    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-certs\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.107589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.107571    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-data\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.107939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.107912    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d294-abf2-4b68-bec1-f29971fca466-istio-podinfo\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.108227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.108196    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-istio-token\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.108412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.108393    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-socket\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.209844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.209809    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-istio-token\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.209871    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-socket\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.209959    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d294-abf2-4b68-bec1-f29971fca466-istiod-ca-cert\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.209995    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-credential-socket\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210022    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwzv\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-kube-api-access-xzwzv\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210064    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-envoy\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210158    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-certs\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210237    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-data\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210378    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-socket\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210396    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d294-abf2-4b68-bec1-f29971fca466-istio-podinfo\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210764    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-certs\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210814    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-data\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.210951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.210893    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-credential-socket\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.227255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.213155    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-envoy\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.227255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.214058    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d294-abf2-4b68-bec1-f29971fca466-istiod-ca-cert\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.227255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.214578    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d294-abf2-4b68-bec1-f29971fca466-istio-podinfo\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.248488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.241368    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-istio-token\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.278850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.278779    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwzv\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-kube-api-access-xzwzv\") pod \"e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") " pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.419226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.419192    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:21.564445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.564235    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main" containerID="cri-o://05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742" gracePeriod=45
Oct 30 01:23:21.964007 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:23:21.963968    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd41d294_abf2_4b68_bec1_f29971fca466.slice/crio-dc71a96b78ff292046803b9047e0b7968b8d8be10a8b661d89c198c66074a161 WatchSource:0}: Error finding container dc71a96b78ff292046803b9047e0b7968b8d8be10a8b661d89c198c66074a161: Status 404 returned error can't find the container with id dc71a96b78ff292046803b9047e0b7968b8d8be10a8b661d89c198c66074a161
Oct 30 01:23:21.966923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.966893    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"]
Oct 30 01:23:21.969297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.968599    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:21.969297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.968666    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:21.969297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:21.968700    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:22.044083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.044045    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:23:22.143352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.143313    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-dshm\") pod \"93ca880f-cacc-42b7-9250-429c367c4f15\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") "
Oct 30 01:23:22.143692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.143656    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-home\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-home\") pod \"93ca880f-cacc-42b7-9250-429c367c4f15\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") "
Oct 30 01:23:22.144546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.144038    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-home" pod "93ca880f-cacc-42b7-9250-429c367c4f15" (UID: "93ca880f-cacc-42b7-9250-429c367c4f15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:22.144546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.144047    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-model-cache\") pod \"93ca880f-cacc-42b7-9250-429c367c4f15\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") "
Oct 30 01:23:22.144752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.144189    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-tmp-dir\") pod \"93ca880f-cacc-42b7-9250-429c367c4f15\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") "
Oct 30 01:23:22.144870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.144824    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ca880f-cacc-42b7-9250-429c367c4f15-kube-api-access-th4d5\" (UniqueName: \"kubernetes.io/projected/93ca880f-cacc-42b7-9250-429c367c4f15-kube-api-access-th4d5\") pod \"93ca880f-cacc-42b7-9250-429c367c4f15\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") "
Oct 30 01:23:22.144944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.144917    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ca880f-cacc-42b7-9250-429c367c4f15-tls-certs\" (UniqueName: \"kubernetes.io/secret/93ca880f-cacc-42b7-9250-429c367c4f15-tls-certs\") pod \"93ca880f-cacc-42b7-9250-429c367c4f15\" (UID: \"93ca880f-cacc-42b7-9250-429c367c4f15\") "
Oct 30 01:23:22.145537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.144276    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-model-cache" pod "93ca880f-cacc-42b7-9250-429c367c4f15" (UID: "93ca880f-cacc-42b7-9250-429c367c4f15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:22.145537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.144426    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-tmp-dir" pod "93ca880f-cacc-42b7-9250-429c367c4f15" (UID: "93ca880f-cacc-42b7-9250-429c367c4f15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:22.146224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.146155    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.146224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.146186    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.146224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.146202    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.147130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.146931    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-dshm" pod "93ca880f-cacc-42b7-9250-429c367c4f15" (UID: "93ca880f-cacc-42b7-9250-429c367c4f15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:22.148373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.148278    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ca880f-cacc-42b7-9250-429c367c4f15-kube-api-access-th4d5" pod "93ca880f-cacc-42b7-9250-429c367c4f15" (UID: "93ca880f-cacc-42b7-9250-429c367c4f15"). InnerVolumeSpecName "kube-api-access-th4d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:22.149738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.149707    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ca880f-cacc-42b7-9250-429c367c4f15-tls-certs" pod "93ca880f-cacc-42b7-9250-429c367c4f15" (UID: "93ca880f-cacc-42b7-9250-429c367c4f15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:22.250952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.248121    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ca880f-cacc-42b7-9250-429c367c4f15-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.250952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.248159    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th4d5\" (UniqueName: \"kubernetes.io/projected/93ca880f-cacc-42b7-9250-429c367c4f15-kube-api-access-th4d5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.250952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.248175    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ca880f-cacc-42b7-9250-429c367c4f15-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.517362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.517263    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" event={"ID":"bd41d294-abf2-4b68-bec1-f29971fca466","Type":"ContainerStarted","Data":"b2002898df8c659365fc2e8744c05f4c17b9065db867f550e7a4a3826d6a7544"}
Oct 30 01:23:22.517362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.517334    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" event={"ID":"bd41d294-abf2-4b68-bec1-f29971fca466","Type":"ContainerStarted","Data":"dc71a96b78ff292046803b9047e0b7968b8d8be10a8b661d89c198c66074a161"}
Oct 30 01:23:22.519987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.519954    5104 generic.go:356] "Generic (PLEG): container finished" podID="93ca880f-cacc-42b7-9250-429c367c4f15" containerID="05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742" exitCode=0
Oct 30 01:23:22.520164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.520014    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"
Oct 30 01:23:22.520164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.520063    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" event={"ID":"93ca880f-cacc-42b7-9250-429c367c4f15","Type":"ContainerDied","Data":"05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742"}
Oct 30 01:23:22.520164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.520112    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp" event={"ID":"93ca880f-cacc-42b7-9250-429c367c4f15","Type":"ContainerDied","Data":"35873ca080c744aa1bb911d102bc093fb296cadce86d1b22981e678e09fc2330"}
Oct 30 01:23:22.520164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.520150    5104 scope.go:122] "RemoveContainer" containerID="05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742"
Oct 30 01:23:22.534577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.534551    5104 scope.go:122] "RemoveContainer" containerID="05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742"
Oct 30 01:23:22.534907 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:22.534883    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742\": container with ID starting with 05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742 not found: ID does not exist" containerID="05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742"
Oct 30 01:23:22.535020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.534915    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742"} err="failed to get container status \"05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742\": rpc error: code = NotFound desc = could not find container \"05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742\": container with ID starting with 05a16fe7e11e83981b13dae49d2183848dcc06aa68444e8309992ba774fda742 not found: ID does not exist"
Oct 30 01:23:22.584484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:22.578644    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podStartSLOduration=2.578621097 podStartE2EDuration="2.578621097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:22.55890758 +0000 UTC m=+2444.106598077" watchObservedRunningTime="2026-10-30 01:23:22.578621097 +0000 UTC m=+2444.126311594"
Oct 30 01:23:23.265644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.265592    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" containerID="cri-o://df8040a68d12757b9fd8909a920ad3ad75ede73fcaa20989280553fd4116e726" gracePeriod=45
Oct 30 01:23:23.284929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.284892    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"]
Oct 30 01:23:23.293890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.293854    5104 cpu_manager.go:414] "containerMap: removing container" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main"
Oct 30 01:23:23.294121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.294105    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main"
Oct 30 01:23:23.302403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.302018    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" containerName="main"
Oct 30 01:23:23.316111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.316080    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"]
Oct 30 01:23:23.316366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.316351    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"]
Oct 30 01:23:23.316608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.316593    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.330866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.330817    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-18a189/auth-test-model-18a189-kserve-7468b8f6df-dlqtp"]
Oct 30 01:23:23.334211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.334176    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73831d-dockercfg-c269j\""
Oct 30 01:23:23.334362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.334258    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73831d\""
Oct 30 01:23:23.396060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.396022    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24729d4-d473-4c88-bf24-1a337be196ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.396361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.396339    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4lw\" (UniqueName: \"kubernetes.io/projected/c24729d4-d473-4c88-bf24-1a337be196ee-kube-api-access-kb4lw\") pod \"payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.420527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.420161    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:23.426947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.426916    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:23.498854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.498813    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24729d4-d473-4c88-bf24-1a337be196ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.500007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.499143    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4lw\" (UniqueName: \"kubernetes.io/projected/c24729d4-d473-4c88-bf24-1a337be196ee-kube-api-access-kb4lw\") pod \"payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.500007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.499929    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24729d4-d473-4c88-bf24-1a337be196ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.525765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.525668    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4lw\" (UniqueName: \"kubernetes.io/projected/c24729d4-d473-4c88-bf24-1a337be196ee-kube-api-access-kb4lw\") pod \"payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.536213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.536181    5104 generic.go:356] "Generic (PLEG): container finished" podID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerID="df8040a68d12757b9fd8909a920ad3ad75ede73fcaa20989280553fd4116e726" exitCode=0
Oct 30 01:23:23.536416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.536282    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" event={"ID":"88a3c713-4ccd-4243-98ac-59b9e73b2823","Type":"ContainerDied","Data":"df8040a68d12757b9fd8909a920ad3ad75ede73fcaa20989280553fd4116e726"}
Oct 30 01:23:23.539075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.539046    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:23.540043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.540022    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:23:23.668090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.668063    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:23:23.707540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.702657    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a3c713-4ccd-4243-98ac-59b9e73b2823-kube-api-access-cd6h4\" (UniqueName: \"kubernetes.io/projected/88a3c713-4ccd-4243-98ac-59b9e73b2823-kube-api-access-cd6h4\") pod \"88a3c713-4ccd-4243-98ac-59b9e73b2823\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") "
Oct 30 01:23:23.707540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.702804    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-dshm\") pod \"88a3c713-4ccd-4243-98ac-59b9e73b2823\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") "
Oct 30 01:23:23.707540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.702918    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-tmp-dir\") pod \"88a3c713-4ccd-4243-98ac-59b9e73b2823\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") "
Oct 30 01:23:23.707540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.702948    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-home\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-home\") pod \"88a3c713-4ccd-4243-98ac-59b9e73b2823\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") "
Oct 30 01:23:23.707540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.703051    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-model-cache\") pod \"88a3c713-4ccd-4243-98ac-59b9e73b2823\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") "
Oct 30 01:23:23.707540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.703078    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a3c713-4ccd-4243-98ac-59b9e73b2823-tls-certs\" (UniqueName: \"kubernetes.io/secret/88a3c713-4ccd-4243-98ac-59b9e73b2823-tls-certs\") pod \"88a3c713-4ccd-4243-98ac-59b9e73b2823\" (UID: \"88a3c713-4ccd-4243-98ac-59b9e73b2823\") "
Oct 30 01:23:23.707540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.704112    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:23.711221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.710384    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-home" pod "88a3c713-4ccd-4243-98ac-59b9e73b2823" (UID: "88a3c713-4ccd-4243-98ac-59b9e73b2823"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.711221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.710616    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-tmp-dir" pod "88a3c713-4ccd-4243-98ac-59b9e73b2823" (UID: "88a3c713-4ccd-4243-98ac-59b9e73b2823"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.711221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.710832    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-model-cache" pod "88a3c713-4ccd-4243-98ac-59b9e73b2823" (UID: "88a3c713-4ccd-4243-98ac-59b9e73b2823"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.752302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.752228    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-dshm" pod "88a3c713-4ccd-4243-98ac-59b9e73b2823" (UID: "88a3c713-4ccd-4243-98ac-59b9e73b2823"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:23.758860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.758796    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a3c713-4ccd-4243-98ac-59b9e73b2823-kube-api-access-cd6h4" pod "88a3c713-4ccd-4243-98ac-59b9e73b2823" (UID: "88a3c713-4ccd-4243-98ac-59b9e73b2823"). InnerVolumeSpecName "kube-api-access-cd6h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:23.759610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.759577    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a3c713-4ccd-4243-98ac-59b9e73b2823-tls-certs" pod "88a3c713-4ccd-4243-98ac-59b9e73b2823" (UID: "88a3c713-4ccd-4243-98ac-59b9e73b2823"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:23.835911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.830297    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6h4\" (UniqueName: \"kubernetes.io/projected/88a3c713-4ccd-4243-98ac-59b9e73b2823-kube-api-access-cd6h4\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.835911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.830344    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.835911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.830362    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.835911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.830375    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.835911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.830386    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a3c713-4ccd-4243-98ac-59b9e73b2823-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.835911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:23.830398    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a3c713-4ccd-4243-98ac-59b9e73b2823-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:24.261143 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:23:24.261112    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24729d4_d473_4c88_bf24_1a337be196ee.slice/crio-e8b0128f6f8bbc6ff8c9d97fd8c8b47dfc0bbfff8266e0821fc01bc0146ace30 WatchSource:0}: Error finding container e8b0128f6f8bbc6ff8c9d97fd8c8b47dfc0bbfff8266e0821fc01bc0146ace30: Status 404 returned error can't find the container with id e8b0128f6f8bbc6ff8c9d97fd8c8b47dfc0bbfff8266e0821fc01bc0146ace30
Oct 30 01:23:24.261383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.261352    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"]
Oct 30 01:23:24.550927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.550822    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5" event={"ID":"c24729d4-d473-4c88-bf24-1a337be196ee","Type":"ContainerStarted","Data":"fc64a3b2fa6f39ea8db351b84c741c978a517f3d3bb719bdee3ce1def5f47023"}
Oct 30 01:23:24.550927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.550883    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:24.550927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.550902    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5" event={"ID":"c24729d4-d473-4c88-bf24-1a337be196ee","Type":"ContainerStarted","Data":"e8b0128f6f8bbc6ff8c9d97fd8c8b47dfc0bbfff8266e0821fc01bc0146ace30"}
Oct 30 01:23:24.552955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.552931    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"
Oct 30 01:23:24.553108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.552972    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" event={"ID":"88a3c713-4ccd-4243-98ac-59b9e73b2823","Type":"ContainerDied","Data":"53cfd23e6b5620a21aa8021e0cd8174f178c434981840b246de31785ac503ddc"}
Oct 30 01:23:24.553108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.553020    5104 scope.go:122] "RemoveContainer" containerID="df8040a68d12757b9fd8909a920ad3ad75ede73fcaa20989280553fd4116e726"
Oct 30 01:23:24.583875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.583840    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ca880f-cacc-42b7-9250-429c367c4f15" path="/var/lib/kubelet/pods/93ca880f-cacc-42b7-9250-429c367c4f15/volumes"
Oct 30 01:23:24.588678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.588630    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5" podStartSLOduration=2.588616775 podStartE2EDuration="2.588616775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:24.587762348 +0000 UTC m=+2446.135452841" watchObservedRunningTime="2026-10-30 01:23:24.588616775 +0000 UTC m=+2446.136307272"
Oct 30 01:23:24.603767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.603726    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main" probeResult="failure" output="Get \"https://10.132.1.36:8000/ready\": context deadline exceeded"
Oct 30 01:23:24.626152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.626108    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"]
Oct 30 01:23:24.631578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:24.631551    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-262230/rate-test-model-262230-kserve-955d7d757-f4ld2"]
Oct 30 01:23:25.235823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:25.235783    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:23:26.588890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:26.588857    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" path="/var/lib/kubelet/pods/88a3c713-4ccd-4243-98ac-59b9e73b2823/volumes"
Oct 30 01:23:27.392571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:27.392491    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"]
Oct 30 01:23:27.553509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:27.538235    5104 cpu_manager.go:414] "containerMap: removing container" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main"
Oct 30 01:23:27.553509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:27.538275    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main"
Oct 30 01:23:27.553509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:27.540397    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a3c713-4ccd-4243-98ac-59b9e73b2823" containerName="main"
Oct 30 01:23:28.047481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.046689    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.076744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.076682    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.098886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.098844    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:28.143845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.143782    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.349538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.345694    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:23:28.372111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.355897    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.424473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.424425    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d3ed60\""
Oct 30 01:23:28.444897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.444860    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d3ed60-dockercfg-5d6s2\""
Oct 30 01:23:28.476809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.476771    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"]
Oct 30 01:23:28.568216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.568169    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615c0cab-adfe-4284-83a6-63b0dfaaf811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:28.576401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.568516    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4tk\" (UniqueName: \"kubernetes.io/projected/615c0cab-adfe-4284-83a6-63b0dfaaf811-kube-api-access-pp4tk\") pod \"payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:28.597672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.596097    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.611695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.611579    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.638494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.634560    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.645082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.645024    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.719796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.719758    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"]
Oct 30 01:23:28.770788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.769820    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.779485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.773240    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"]
Oct 30 01:23:28.779485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.774279    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:28.779485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.777129    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615c0cab-adfe-4284-83a6-63b0dfaaf811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:28.779485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.777222    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4tk\" (UniqueName: \"kubernetes.io/projected/615c0cab-adfe-4284-83a6-63b0dfaaf811-kube-api-access-pp4tk\") pod \"payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:28.779485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.778687    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615c0cab-adfe-4284-83a6-63b0dfaaf811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:28.837600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.836570    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:28.875502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.875358    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4tk\" (UniqueName: \"kubernetes.io/projected/615c0cab-adfe-4284-83a6-63b0dfaaf811-kube-api-access-pp4tk\") pod \"payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:28.884108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.880028    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fhjf\" (UniqueName: \"kubernetes.io/projected/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-kube-api-access-9fhjf\") pod \"payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:28.884108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.880385    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:28.884108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.880695    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 30 01:23:28.901761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:28.901726    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:29.006879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.006831    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhjf\" (UniqueName: \"kubernetes.io/projected/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-kube-api-access-9fhjf\") pod \"payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:29.044484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.007399    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:29.080934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.058815    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:29.080934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.058879    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:29.080934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.075224    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:23:29.084929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.082009    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:29.084929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.082660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:23:29.167525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.163603    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:29.167525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.164187    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:23:29.291500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.256750    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fhjf\" (UniqueName: \"kubernetes.io/projected/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-kube-api-access-9fhjf\") pod \"payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:29.397498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.394157    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:23:29.397498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.395630    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:23:29.414310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.413918    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:29.441538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.414645    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:23:29.589674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.589349    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:29.616097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.614796    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:29.839608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.634051    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:23:29.841804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.634401    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:29.845542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.678016    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:23:29.846447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.846413    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:23:29.846720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.846703    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:23:29.846892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:29.694572    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.000084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.000036    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.094666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.094561    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.227483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.171365    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:23:30.227796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.227765    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"]
Oct 30 01:23:30.249681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.174559    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.263525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.203765    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.415608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.414524    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.415608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.414729    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.625710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.625646    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Oct 30 01:23:30.780123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.710023    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.882786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.882733    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:30.900597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.889035    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:23:30.900597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.889306    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:23:31.008481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.996843    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:23:31.008481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:30.997003    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"]
Oct 30 01:23:31.008481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.004208    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:31.013624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.013592    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:31.066527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.065060    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4be164-dockercfg-bdrv9\""
Oct 30 01:23:31.066527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.065172    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4be164\""
Oct 30 01:23:31.633625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.615599    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:31.633625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.623138    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:31.677026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.672770    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:31.758381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.722442    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:23:31.758381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.724057    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:31.949482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:31.934385    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:23:32.054483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:32.044403    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:23:32.577658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:32.577603    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:23:32.711896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:32.711839    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:32.793389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:32.793344    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:23:33.259942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.259896    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228613f-dded-4bb1-90e0-6f918e4f58f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:33.260153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.260013    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82h6c\" (UniqueName: \"kubernetes.io/projected/b228613f-dded-4bb1-90e0-6f918e4f58f9-kube-api-access-82h6c\") pod \"payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:33.260417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.260360    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj" podUID="b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.260643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.138813    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded"
Oct 30 01:23:33.261614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.153660    5104 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-wfw4k container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": context deadline exceeded" start-of-body=
Oct 30 01:23:33.261802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.261771    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wfw4k" podUID="7a355cec-6892-4b36-89b7-60c6b3460bf0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": context deadline exceeded"
Oct 30 01:23:33.261943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.161640    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.187:50052\" within 1s: context deadline exceeded"
Oct 30 01:23:33.262062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.165848    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z2ktb" podUID="626e0a50-e9f7-4b0f-aa09-226cba9a0dfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.262552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.262524    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:23:33.263438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.159064    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz" podUID="955ad4ba-5e31-4758-8666-744d4549da0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.263597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.202797    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.264639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.202897    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:23:33.264639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.264630    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.264794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.195215    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:23:33.275157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.275107    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9" podUID="96d0232a-84d5-44ff-b2b6-d45f390c392d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.278158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.277565    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h" podUID="4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.171:50052\" within 1s: context deadline exceeded"
Oct 30 01:23:33.303746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.014650    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" podUID="439a5722-bfae-47e0-b389-c682c528531a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded"
Oct 30 01:23:33.307471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.075740    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.188:50052\" within 1s: context deadline exceeded"
Oct 30 01:23:33.307471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.138730    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:23:33.331325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.328096    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.221:50052\" within 1s: context deadline exceeded"
Oct 30 01:23:33.331325 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:23:33.329388    5104 logging.go:55] [core] [Channel #11024 SubChannel #11025]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.171:50052", ServerName: "10.132.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:49064->10.132.0.171:50052: use of closed network connection"
Oct 30 01:23:33.335967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.334694    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.350726 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:33.344858    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.348s"
Oct 30 01:23:33.350726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.347945    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:33.351896 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:23:33.351869    5104 logging.go:55] [core] [Channel #11023 SubChannel #11039]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.221:50052", ServerName: "10.132.0.221:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:58980->10.132.0.221:50052: use of closed network connection"
Oct 30 01:23:33.361310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.361251    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:23:33.362662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.362434    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228613f-dded-4bb1-90e0-6f918e4f58f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:33.362918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.362897    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82h6c\" (UniqueName: \"kubernetes.io/projected/b228613f-dded-4bb1-90e0-6f918e4f58f9-kube-api-access-82h6c\") pod \"payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:33.364571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.364533    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228613f-dded-4bb1-90e0-6f918e4f58f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:33.372388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.372353    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:23:33.386603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.372418    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:23:33.386603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.372473    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:23:33.386603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.375244    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:23:33.408819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.408786    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:23:33.429049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.424311    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82h6c\" (UniqueName: \"kubernetes.io/projected/b228613f-dded-4bb1-90e0-6f918e4f58f9-kube-api-access-82h6c\") pod \"payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:33.429049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.424451    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:23:33.429871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.429846    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:23:33.435776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.435135    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:23:33.445876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.445821    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:23:33.451091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.451063    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:23:33.523444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.523403    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"]
Oct 30 01:23:33.543534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.543507    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"]
Oct 30 01:23:33.552606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.552577    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149f6257-6ddrv"]
Oct 30 01:23:33.565962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.565935    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"]
Oct 30 01:23:33.669899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.669864    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"]
Oct 30 01:23:33.672292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.672155    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:33.726184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.726153    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"]
Oct 30 01:23:33.726750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.726287    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.744202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.744044    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:23:33.744202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.744168    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:23:33.807670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.784857    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:23:33.807670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.784938    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:33.820100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.819021    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4sz2\" (UniqueName: \"kubernetes.io/projected/a5555030-b09d-4df4-9632-7575b8d2f19f-kube-api-access-c4sz2\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.820100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.819211    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.820100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.819366    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.922303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.922265    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sz2\" (UniqueName: \"kubernetes.io/projected/a5555030-b09d-4df4-9632-7575b8d2f19f-kube-api-access-c4sz2\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.922525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.922473    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.923022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.922641    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.925421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.925394    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.928753 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.928724    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:33.950075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:33.950033    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4sz2\" (UniqueName: \"kubernetes.io/projected/a5555030-b09d-4df4-9632-7575b8d2f19f-kube-api-access-c4sz2\") pod \"maas-api-revoke-sub-a36e0284-cqt4n\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:34.145602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.145559    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:34.161187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.161127    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"]
Oct 30 01:23:34.420268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.420229    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph" event={"ID":"b228613f-dded-4bb1-90e0-6f918e4f58f9","Type":"ContainerStarted","Data":"065ee11c8a39b8f07ffcd5fd89ca37b947f042d3b935a441589138a465e6573c"}
Oct 30 01:23:34.423649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.423619    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct" event={"ID":"615c0cab-adfe-4284-83a6-63b0dfaaf811","Type":"ContainerStarted","Data":"272e47c14528a86d755f779504f80cdc707d9796e16358f44bb75cedad0f0d12"}
Oct 30 01:23:34.423920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.423903    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:34.424200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.424181    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct" event={"ID":"615c0cab-adfe-4284-83a6-63b0dfaaf811","Type":"ContainerStarted","Data":"bba0a790ea545b661330b780ce69ddb1d345f5be4a7f2f18559c84c762ce37bb"}
Oct 30 01:23:34.427504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.426737    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4" event={"ID":"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6","Type":"ContainerStarted","Data":"5fd4979d6539f84f46c17414c2bc7ec0895c11c75c72b47add24062a2fe386fc"}
Oct 30 01:23:34.427504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.426783    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4" event={"ID":"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6","Type":"ContainerStarted","Data":"16132d0453fc1dc28193505795566474208f86b79ecc3f1114ef531cb6188d38"}
Oct 30 01:23:34.427504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.426934    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:34.465617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.465397    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct" podStartSLOduration=8.46538124 podStartE2EDuration="8.46538124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:34.4598402 +0000 UTC m=+2456.007530696" watchObservedRunningTime="2026-10-30 01:23:34.46538124 +0000 UTC m=+2456.013071726"
Oct 30 01:23:34.650595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.650450    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843d580a-b98b-4426-8ac7-844bb4ddf3bb" path="/var/lib/kubelet/pods/843d580a-b98b-4426-8ac7-844bb4ddf3bb/volumes"
Oct 30 01:23:34.709014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.708946    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4" podStartSLOduration=7.708919188 podStartE2EDuration="7.708919188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:34.508909531 +0000 UTC m=+2456.056600029" watchObservedRunningTime="2026-10-30 01:23:34.708919188 +0000 UTC m=+2456.256609687"
Oct 30 01:23:34.718225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:34.718177    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"]
Oct 30 01:23:35.040822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.040781    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" containerID="cri-o://298696350ecebe95e009b91b958600ab85c621d2e5e3575258e6221fdfab4fa4" gracePeriod=45
Oct 30 01:23:35.051745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.051700    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": dial tcp 10.132.1.41:8000: connect: connection refused"
Oct 30 01:23:35.449227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.449194    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph" event={"ID":"b228613f-dded-4bb1-90e0-6f918e4f58f9","Type":"ContainerStarted","Data":"703ced038ee97f8f0dcd6050b0ef11cc7930f6d4c491089f31c85d12f6010bc7"}
Oct 30 01:23:35.449366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.449326    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:35.451143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.451120    5104 generic.go:356] "Generic (PLEG): container finished" podID="7caba728-a591-4de9-a400-44378f60f7c7" containerID="298696350ecebe95e009b91b958600ab85c621d2e5e3575258e6221fdfab4fa4" exitCode=0
Oct 30 01:23:35.451228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.451189    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" event={"ID":"7caba728-a591-4de9-a400-44378f60f7c7","Type":"ContainerDied","Data":"298696350ecebe95e009b91b958600ab85c621d2e5e3575258e6221fdfab4fa4"}
Oct 30 01:23:35.455927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.455899    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n" event={"ID":"a5555030-b09d-4df4-9632-7575b8d2f19f","Type":"ContainerStarted","Data":"8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f"}
Oct 30 01:23:35.460294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.460271    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:23:35.552486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.539202    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph" podStartSLOduration=6.539179268 podStartE2EDuration="6.539179268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:35.528238761 +0000 UTC m=+2457.075929257" watchObservedRunningTime="2026-10-30 01:23:35.539179268 +0000 UTC m=+2457.086869760"
Oct 30 01:23:35.680532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.668361    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-model-cache\") pod \"7caba728-a591-4de9-a400-44378f60f7c7\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") "
Oct 30 01:23:35.680532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.668562    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-dshm\") pod \"7caba728-a591-4de9-a400-44378f60f7c7\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") "
Oct 30 01:23:35.680532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.668597    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7caba728-a591-4de9-a400-44378f60f7c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/7caba728-a591-4de9-a400-44378f60f7c7-tls-certs\") pod \"7caba728-a591-4de9-a400-44378f60f7c7\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") "
Oct 30 01:23:35.680532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.668652    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-tmp-dir\") pod \"7caba728-a591-4de9-a400-44378f60f7c7\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") "
Oct 30 01:23:35.680532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.668713    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-home\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-home\") pod \"7caba728-a591-4de9-a400-44378f60f7c7\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") "
Oct 30 01:23:35.680532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.668894    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7caba728-a591-4de9-a400-44378f60f7c7-kube-api-access-7sbr2\" (UniqueName: \"kubernetes.io/projected/7caba728-a591-4de9-a400-44378f60f7c7-kube-api-access-7sbr2\") pod \"7caba728-a591-4de9-a400-44378f60f7c7\" (UID: \"7caba728-a591-4de9-a400-44378f60f7c7\") "
Oct 30 01:23:35.727612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.727567    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-model-cache" pod "7caba728-a591-4de9-a400-44378f60f7c7" (UID: "7caba728-a591-4de9-a400-44378f60f7c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.865501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.762089    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-home" pod "7caba728-a591-4de9-a400-44378f60f7c7" (UID: "7caba728-a591-4de9-a400-44378f60f7c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.865501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.782213    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-tmp-dir" pod "7caba728-a591-4de9-a400-44378f60f7c7" (UID: "7caba728-a591-4de9-a400-44378f60f7c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.865501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.770934    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:23:35.865501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.860184    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n" podStartSLOduration=2.860159915 podStartE2EDuration="2.860159915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:35.62083135 +0000 UTC m=+2457.168521847" watchObservedRunningTime="2026-10-30 01:23:35.860159915 +0000 UTC m=+2457.407850412"
Oct 30 01:23:35.892901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.888613    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.892901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.888646    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.892901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.888661    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.926702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.926662    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caba728-a591-4de9-a400-44378f60f7c7-kube-api-access-7sbr2" pod "7caba728-a591-4de9-a400-44378f60f7c7" (UID: "7caba728-a591-4de9-a400-44378f60f7c7"). InnerVolumeSpecName "kube-api-access-7sbr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:35.958690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.958609    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7caba728-a591-4de9-a400-44378f60f7c7-tls-certs" pod "7caba728-a591-4de9-a400-44378f60f7c7" (UID: "7caba728-a591-4de9-a400-44378f60f7c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:35.959565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.959523    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-dshm" pod "7caba728-a591-4de9-a400-44378f60f7c7" (UID: "7caba728-a591-4de9-a400-44378f60f7c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.991084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.991046    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sbr2\" (UniqueName: \"kubernetes.io/projected/7caba728-a591-4de9-a400-44378f60f7c7-kube-api-access-7sbr2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.991319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.991303    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caba728-a591-4de9-a400-44378f60f7c7-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.991425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:35.991413    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caba728-a591-4de9-a400-44378f60f7c7-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:36.046484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.025976    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"]
Oct 30 01:23:36.046484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.045159    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main"
Oct 30 01:23:36.046484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.045196    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main"
Oct 30 01:23:36.046484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.045559    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7caba728-a591-4de9-a400-44378f60f7c7" containerName="main"
Oct 30 01:23:36.061915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.061882    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.068870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.068837    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"]
Oct 30 01:23:36.129150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.128726    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"]
Oct 30 01:23:36.129150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.128893    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.144061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.144031    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"]
Oct 30 01:23:36.230375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230314    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.230575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230383    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.230575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230499    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.230575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230550    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.230796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230582    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbfm\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-kube-api-access-tqbfm\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.230796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230641    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.230950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230920    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.231017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.230959    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jdm\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-kube-api-access-w5jdm\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.231117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231092    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.231177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231127    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79636883-5bc5-478e-a99f-50df03195498-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.231177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231145    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.231291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231251    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.231355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231321    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79636883-5bc5-478e-a99f-50df03195498-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.231413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231380    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.231514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231496    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.231556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231534    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.231599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231571    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a0bc92-3340-43a0-811c-db63789c6a05-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.231639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.231629    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333057    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333266    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.333310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333295    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jdm\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-kube-api-access-w5jdm\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333386    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.333431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333418    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79636883-5bc5-478e-a99f-50df03195498-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333447    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.333569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333514    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333580    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79636883-5bc5-478e-a99f-50df03195498-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333635    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333713    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.333794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333741    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.333794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333779    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a0bc92-3340-43a0-811c-db63789c6a05-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.333971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333844    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333912    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.333971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333940    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.334135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.333995    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.334135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334029    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.334135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334059    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbfm\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-kube-api-access-tqbfm\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.334295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334180    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.334358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334299    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.334563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334535    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.334732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334708    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.334835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334632    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.334925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334819    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a0bc92-3340-43a0-811c-db63789c6a05-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.335018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.334909    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.335115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.335092    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.335179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.335134    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.336028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.336001    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79636883-5bc5-478e-a99f-50df03195498-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.336756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.336709    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.336873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.336849    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.337831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.337803    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.338074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.338048    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79636883-5bc5-478e-a99f-50df03195498-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.353084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.353056    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.353495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.353452    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.353684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.353662    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbfm\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-kube-api-access-tqbfm\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.353804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.353788    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jdm\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-kube-api-access-w5jdm\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.437525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.437484    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:36.455576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.455547    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:36.469959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.469884    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"
Oct 30 01:23:36.469959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.469900    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695" event={"ID":"7caba728-a591-4de9-a400-44378f60f7c7","Type":"ContainerDied","Data":"b8123f12364763fc3211d53cfa96418ff5a4c496a6daacfb82cd69546385efed"}
Oct 30 01:23:36.470144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.469962    5104 scope.go:122] "RemoveContainer" containerID="298696350ecebe95e009b91b958600ab85c621d2e5e3575258e6221fdfab4fa4"
Oct 30 01:23:36.472215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.472184    5104 generic.go:356] "Generic (PLEG): container finished" podID="a5555030-b09d-4df4-9632-7575b8d2f19f" containerID="e8e48cfb0b7b93c05e12994297636b27d7d27633d95fc072a9f0818369b5278b" exitCode=0
Oct 30 01:23:36.472359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.472290    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n" event={"ID":"a5555030-b09d-4df4-9632-7575b8d2f19f","Type":"ContainerDied","Data":"e8e48cfb0b7b93c05e12994297636b27d7d27633d95fc072a9f0818369b5278b"}
Oct 30 01:23:36.659192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.659163    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"]
Oct 30 01:23:36.659410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.659395    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-add1bb/rate-test-model-add1bb-kserve-6664bd5965-hq695"]
Oct 30 01:23:36.825885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.825840    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"]
Oct 30 01:23:36.826125 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:23:36.825897    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a0bc92_3340_43a0_811c_db63789c6a05.slice/crio-8c6b1cb5973cce72c005f0f24badcbf4e2c1902296ded31f3c82d0d0779f578e WatchSource:0}: Error finding container 8c6b1cb5973cce72c005f0f24badcbf4e2c1902296ded31f3c82d0d0779f578e: Status 404 returned error can't find the container with id 8c6b1cb5973cce72c005f0f24badcbf4e2c1902296ded31f3c82d0d0779f578e
Oct 30 01:23:36.831562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.830729    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:36.831562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.830825    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:36.831562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:36.830872    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:37.281225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.281181    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"]
Oct 30 01:23:37.355426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.355380    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:37.355572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.355489    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:37.355572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.355530    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:37.609578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.602195    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" event={"ID":"79636883-5bc5-478e-a99f-50df03195498","Type":"ContainerStarted","Data":"d6dab1f78146281d2338ef6ed3ce6891617d42e4e615ce1ab5b4bd5ca77c8ee0"}
Oct 30 01:23:37.609578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.605974    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" event={"ID":"c5a0bc92-3340-43a0-811c-db63789c6a05","Type":"ContainerStarted","Data":"cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced"}
Oct 30 01:23:37.609578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.606029    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" event={"ID":"c5a0bc92-3340-43a0-811c-db63789c6a05","Type":"ContainerStarted","Data":"8c6b1cb5973cce72c005f0f24badcbf4e2c1902296ded31f3c82d0d0779f578e"}
Oct 30 01:23:37.653283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.648587    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podStartSLOduration=2.64856755 podStartE2EDuration="2.64856755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:37.646852898 +0000 UTC m=+2459.194543395" watchObservedRunningTime="2026-10-30 01:23:37.64856755 +0000 UTC m=+2459.196258049"
Oct 30 01:23:37.760511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.760155    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"]
Oct 30 01:23:37.876500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:37.875614    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"]
Oct 30 01:23:38.097057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.097017    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:38.284516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.276099    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5555030-b09d-4df4-9632-7575b8d2f19f-kube-api-access-c4sz2\" (UniqueName: \"kubernetes.io/projected/a5555030-b09d-4df4-9632-7575b8d2f19f-kube-api-access-c4sz2\") pod \"a5555030-b09d-4df4-9632-7575b8d2f19f\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") "
Oct 30 01:23:38.539512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.538344    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:38.539512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.539196    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:38.539512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.539253    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:38.892747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.889844    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-openshift-ca\") pod \"a5555030-b09d-4df4-9632-7575b8d2f19f\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") "
Oct 30 01:23:38.892747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.890286    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\") pod \"a5555030-b09d-4df4-9632-7575b8d2f19f\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") "
Oct 30 01:23:38.910228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:38.910173    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5555030-b09d-4df4-9632-7575b8d2f19f-kube-api-access-c4sz2" pod "a5555030-b09d-4df4-9632-7575b8d2f19f" (UID: "a5555030-b09d-4df4-9632-7575b8d2f19f"). InnerVolumeSpecName "kube-api-access-c4sz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:39.201522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.074451    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-openshift-ca" pod "a5555030-b09d-4df4-9632-7575b8d2f19f" (UID: "a5555030-b09d-4df4-9632-7575b8d2f19f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:39.288494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.285408    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4sz2\" (UniqueName: \"kubernetes.io/projected/a5555030-b09d-4df4-9632-7575b8d2f19f-kube-api-access-c4sz2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:39.288494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.285447    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:39.431534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.397662    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca" pod "a5555030-b09d-4df4-9632-7575b8d2f19f" (UID: "a5555030-b09d-4df4-9632-7575b8d2f19f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:39.431534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.423336    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\") pod \"a5555030-b09d-4df4-9632-7575b8d2f19f\" (UID: \"a5555030-b09d-4df4-9632-7575b8d2f19f\") "
Oct 30 01:23:39.431534 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:23:39.423498    5104 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a5555030-b09d-4df4-9632-7575b8d2f19f/volumes/kubernetes.io~secret/maas-api-xks-ca
Oct 30 01:23:39.431534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.423617    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca" pod "a5555030-b09d-4df4-9632-7575b8d2f19f" (UID: "a5555030-b09d-4df4-9632-7575b8d2f19f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:39.504980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.504892    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"
Oct 30 01:23:39.506569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.505876    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7caba728-a591-4de9-a400-44378f60f7c7" path="/var/lib/kubelet/pods/7caba728-a591-4de9-a400-44378f60f7c7/volumes"
Oct 30 01:23:39.537478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.517741    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:39.537478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.517815    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:39.638665 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:39.638629    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.03s"
Oct 30 01:23:39.651031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.650998    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n" event={"ID":"a5555030-b09d-4df4-9632-7575b8d2f19f","Type":"ContainerDied","Data":"8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f"}
Oct 30 01:23:39.651288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.642967    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podStartSLOduration=4.642949713 podStartE2EDuration="4.642949713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:39.624124294 +0000 UTC m=+2461.171814792" watchObservedRunningTime="2026-10-30 01:23:39.642949713 +0000 UTC m=+2461.190640209"
Oct 30 01:23:39.651424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.638921    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5555030-b09d-4df4-9632-7575b8d2f19f-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:39.681831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.681770    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f"
Oct 30 01:23:39.682121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:39.682092    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" event={"ID":"79636883-5bc5-478e-a99f-50df03195498","Type":"ContainerStarted","Data":"ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a"}
Oct 30 01:23:39.761771 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:39.761672    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:23:40.474516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:40.463072    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:40.478400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:40.477329    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Oct 30 01:23:40.478400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:40.477391    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Oct 30 01:23:40.497964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:40.479201    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:40.497964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:40.479265    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:41.438691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:41.438651    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:41.438977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:41.438948    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:41.456294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:41.456265    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Oct 30 01:23:41.456516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:41.456496    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Oct 30 01:23:42.457833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:42.457549    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:42.457833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:42.457614    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:42.458415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:42.458094    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Oct 30 01:23:42.458415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:42.458132    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Oct 30 01:23:42.760767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:42.760651    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt" podUID="add97fd5-9780-47df-bb5f-c523d9700c1a" containerName="payload-processing" containerID="cri-o://5356b7dd1e12fdd46191562ac6673fa73e56942f1134fb485051802cc4edf0f1" gracePeriod=25
Oct 30 01:23:42.881512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:42.877295    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m" podUID="559b9e48-a828-4da7-807b-031b99769899" containerName="payload-pre-processing" containerID="cri-o://73df0d98d557de28c1a850a34883a08b5e6080f65fcedea61c60a3c7bb2b2fb4" gracePeriod=25
Oct 30 01:23:43.558604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.485593    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:43.558604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.485664    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:43.559304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.559097    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Oct 30 01:23:43.559304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.559154    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Oct 30 01:23:43.771530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.771446    5104 generic.go:356] "Generic (PLEG): container finished" podID="add97fd5-9780-47df-bb5f-c523d9700c1a" containerID="5356b7dd1e12fdd46191562ac6673fa73e56942f1134fb485051802cc4edf0f1" exitCode=0
Oct 30 01:23:43.771727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.771524    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt" event={"ID":"add97fd5-9780-47df-bb5f-c523d9700c1a","Type":"ContainerDied","Data":"5356b7dd1e12fdd46191562ac6673fa73e56942f1134fb485051802cc4edf0f1"}
Oct 30 01:23:43.771727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.771586    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt" event={"ID":"add97fd5-9780-47df-bb5f-c523d9700c1a","Type":"ContainerDied","Data":"87a47f179352a12f920f41876da08bc173565dddedaaa49c75092c57873deca2"}
Oct 30 01:23:43.771727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.771608    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a47f179352a12f920f41876da08bc173565dddedaaa49c75092c57873deca2"
Oct 30 01:23:43.860669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.860639    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:23:43.868581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.868548    5104 generic.go:356] "Generic (PLEG): container finished" podID="559b9e48-a828-4da7-807b-031b99769899" containerID="73df0d98d557de28c1a850a34883a08b5e6080f65fcedea61c60a3c7bb2b2fb4" exitCode=0
Oct 30 01:23:43.868977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.868953    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m" event={"ID":"559b9e48-a828-4da7-807b-031b99769899","Type":"ContainerDied","Data":"73df0d98d557de28c1a850a34883a08b5e6080f65fcedea61c60a3c7bb2b2fb4"}
Oct 30 01:23:43.872349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.872305    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add97fd5-9780-47df-bb5f-c523d9700c1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add97fd5-9780-47df-bb5f-c523d9700c1a-plugins-config-volume\") pod \"add97fd5-9780-47df-bb5f-c523d9700c1a\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") "
Oct 30 01:23:43.880715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.880685    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add97fd5-9780-47df-bb5f-c523d9700c1a-plugins-config-volume" pod "add97fd5-9780-47df-bb5f-c523d9700c1a" (UID: "add97fd5-9780-47df-bb5f-c523d9700c1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:43.902837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.889596    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add97fd5-9780-47df-bb5f-c523d9700c1a-kube-api-access-4bxtd\" (UniqueName: \"kubernetes.io/projected/add97fd5-9780-47df-bb5f-c523d9700c1a-kube-api-access-4bxtd\") pod \"add97fd5-9780-47df-bb5f-c523d9700c1a\" (UID: \"add97fd5-9780-47df-bb5f-c523d9700c1a\") "
Oct 30 01:23:43.902837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:43.890926    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add97fd5-9780-47df-bb5f-c523d9700c1a-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.256907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.256339    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add97fd5-9780-47df-bb5f-c523d9700c1a-kube-api-access-4bxtd" pod "add97fd5-9780-47df-bb5f-c523d9700c1a" (UID: "add97fd5-9780-47df-bb5f-c523d9700c1a"). InnerVolumeSpecName "kube-api-access-4bxtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:44.359522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.329601    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxtd\" (UniqueName: \"kubernetes.io/projected/add97fd5-9780-47df-bb5f-c523d9700c1a-kube-api-access-4bxtd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.359522 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:44.358718    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache]"
Oct 30 01:23:44.362098 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:44.362052    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache]"
Oct 30 01:23:44.444676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.444627    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:44.444985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.444949    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:44.507771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.500070    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Oct 30 01:23:44.507771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.500151    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Oct 30 01:23:44.512952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.512919    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:23:44.551638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.551595    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"]
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554080    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a5555030-b09d-4df4-9632-7575b8d2f19f" containerName="revoke-keys"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554112    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5555030-b09d-4df4-9632-7575b8d2f19f" containerName="revoke-keys"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554172    5104 cpu_manager.go:414] "containerMap: removing container" podUID="559b9e48-a828-4da7-807b-031b99769899" containerName="payload-pre-processing"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554180    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559b9e48-a828-4da7-807b-031b99769899" containerName="payload-pre-processing"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554199    5104 cpu_manager.go:414] "containerMap: removing container" podUID="add97fd5-9780-47df-bb5f-c523d9700c1a" containerName="payload-processing"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554206    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add97fd5-9780-47df-bb5f-c523d9700c1a" containerName="payload-processing"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554779    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="add97fd5-9780-47df-bb5f-c523d9700c1a" containerName="payload-processing"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554834    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5555030-b09d-4df4-9632-7575b8d2f19f" containerName="revoke-keys"
Oct 30 01:23:44.558287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.554870    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="559b9e48-a828-4da7-807b-031b99769899" containerName="payload-pre-processing"
Oct 30 01:23:44.561513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.561434    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"]
Oct 30 01:23:44.561806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.561788    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.599483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.596849    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:23:44.599483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.596922    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:44.599483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.596943    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:23:44.599483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.596987    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:23:44.662487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.654435    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559b9e48-a828-4da7-807b-031b99769899-kube-api-access-j568g\" (UniqueName: \"kubernetes.io/projected/559b9e48-a828-4da7-807b-031b99769899-kube-api-access-j568g\") pod \"559b9e48-a828-4da7-807b-031b99769899\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") "
Oct 30 01:23:44.662487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.654543    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559b9e48-a828-4da7-807b-031b99769899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/559b9e48-a828-4da7-807b-031b99769899-plugins-config-volume\") pod \"559b9e48-a828-4da7-807b-031b99769899\" (UID: \"559b9e48-a828-4da7-807b-031b99769899\") "
Oct 30 01:23:44.662487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.654930    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a57814a-ee26-4472-862e-a0343b717010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.662487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.655093    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85zfn\" (UniqueName: \"kubernetes.io/projected/9a57814a-ee26-4472-862e-a0343b717010-kube-api-access-85zfn\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.662487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.655314    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a57814a-ee26-4472-862e-a0343b717010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.662915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.662708    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559b9e48-a828-4da7-807b-031b99769899-plugins-config-volume" pod "559b9e48-a828-4da7-807b-031b99769899" (UID: "559b9e48-a828-4da7-807b-031b99769899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:44.672229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.670924    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559b9e48-a828-4da7-807b-031b99769899-kube-api-access-j568g" pod "559b9e48-a828-4da7-807b-031b99769899" (UID: "559b9e48-a828-4da7-807b-031b99769899"). InnerVolumeSpecName "kube-api-access-j568g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:44.757693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.757655    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a57814a-ee26-4472-862e-a0343b717010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.759657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.759545    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a57814a-ee26-4472-862e-a0343b717010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.760052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.760028    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85zfn\" (UniqueName: \"kubernetes.io/projected/9a57814a-ee26-4472-862e-a0343b717010-kube-api-access-85zfn\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.760443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.760422    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j568g\" (UniqueName: \"kubernetes.io/projected/559b9e48-a828-4da7-807b-031b99769899-kube-api-access-j568g\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.763984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.763694    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/559b9e48-a828-4da7-807b-031b99769899-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.763984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.761362    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a57814a-ee26-4472-862e-a0343b717010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.763984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.760489    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a57814a-ee26-4472-862e-a0343b717010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.856485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.853506    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85zfn\" (UniqueName: \"kubernetes.io/projected/9a57814a-ee26-4472-862e-a0343b717010-kube-api-access-85zfn\") pod \"maas-api-revoke-sub-1c09de66-jgfnr\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:44.958514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:44.958480    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:45.045942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.045909    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"
Oct 30 01:23:45.046846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.046824    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"
Oct 30 01:23:45.047497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.047446    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m" event={"ID":"559b9e48-a828-4da7-807b-031b99769899","Type":"ContainerDied","Data":"a9437db512bee6de7306faf20fc538d6117226cf7cef795108220a34bf2f327c"}
Oct 30 01:23:45.047653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.047536    5104 scope.go:122] "RemoveContainer" containerID="73df0d98d557de28c1a850a34883a08b5e6080f65fcedea61c60a3c7bb2b2fb4"
Oct 30 01:23:45.265515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.265186    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"]
Oct 30 01:23:45.279510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.276843    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-6d9b9f68dd-vkxqt"]
Oct 30 01:23:45.345749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.345653    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"]
Oct 30 01:23:45.355151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.355108    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-684c95bf47-9vx7m"]
Oct 30 01:23:45.439482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.438098    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body=
Oct 30 01:23:45.439482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.438170    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused"
Oct 30 01:23:45.465326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.465285    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:23:45.466884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.466842    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Oct 30 01:23:45.467034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.466909    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Oct 30 01:23:45.472406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.472377    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:23:45.562125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.560641    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"]
Oct 30 01:23:45.635619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.630343    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"]
Oct 30 01:23:45.635619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.630550    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.758358    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.758570    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.758634    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.758661    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ec7f91-11f2-43f8-93ce-04b319412bd2-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.758739    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfkk\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-kube-api-access-4vfkk\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.758784    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.758968    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.759016    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.779967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.759158    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.898162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.898106    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.898689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.898632    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.899033 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.899012    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.899323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.899247    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.899323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.899295    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ec7f91-11f2-43f8-93ce-04b319412bd2-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.899752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.899695    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfkk\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-kube-api-access-4vfkk\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.899942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.899893    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.900292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.900246    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.900554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.900441    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.900655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.900590    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.901412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.901384    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.902317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.902278    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.902579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.902558    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.902952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.902910    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ec7f91-11f2-43f8-93ce-04b319412bd2-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.903407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.903384    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.905702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.905653    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.936313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.936234    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"]
Oct 30 01:23:45.938831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.938746    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfkk\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-kube-api-access-4vfkk\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:45.942004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:45.941293    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:46.067470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.067419    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr" event={"ID":"9a57814a-ee26-4472-862e-a0343b717010","Type":"ContainerStarted","Data":"63ae4404cc06e9fe596ceb757092be2435f84531e3c7ce7865c3e66f6588ef37"}
Oct 30 01:23:46.119593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.119557    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:46.443633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.443597    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:46.444196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.444177    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:46.445071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.445049    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:23:46.463793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.463759    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:46.464160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.464136    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:46.465194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.465172    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:23:46.486505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.485659    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:23:46.517677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.517637    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"]
Oct 30 01:23:46.521978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.521928    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:46.522100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.522036    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:46.522100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.522084    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:46.613147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.613106    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559b9e48-a828-4da7-807b-031b99769899" path="/var/lib/kubelet/pods/559b9e48-a828-4da7-807b-031b99769899/volumes"
Oct 30 01:23:46.618414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.618383    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add97fd5-9780-47df-bb5f-c523d9700c1a" path="/var/lib/kubelet/pods/add97fd5-9780-47df-bb5f-c523d9700c1a/volumes"
Oct 30 01:23:46.665086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.664971    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"]
Oct 30 01:23:46.676711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.676674    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.702966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.696744    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e4f8a6-openshift-default-dockercfg-sk5wh\""
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.727583    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-envoy\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.727644    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c066f1b-c608-42af-800c-7167e4c48a80-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.727761    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-data\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.727801    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-istio-token\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.727851    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.727895    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c066f1b-c608-42af-800c-7167e4c48a80-istio-podinfo\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.727981    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-certs\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.728061    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-credential-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.730173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.729611    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwcm\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-kube-api-access-gvwcm\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.753575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.752329    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"]
Oct 30 01:23:46.832557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832253    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwcm\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-kube-api-access-gvwcm\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832580    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-envoy\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832609    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c066f1b-c608-42af-800c-7167e4c48a80-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832704    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-data\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832730    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-istio-token\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832766    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832798    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c066f1b-c608-42af-800c-7167e4c48a80-istio-podinfo\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832874    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-certs\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.832938    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-credential-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.833549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.833425    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-credential-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.837631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.837602    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-envoy\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.837835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.837723    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c066f1b-c608-42af-800c-7167e4c48a80-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.839217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.837959    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-data\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.839217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.838208    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-certs\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.839217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.838730    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.841339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.840908    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c066f1b-c608-42af-800c-7167e4c48a80-istio-podinfo\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.902479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.900560    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-istio-token\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:46.934486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:46.916027    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwcm\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-kube-api-access-gvwcm\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:47.083789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.083670    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" event={"ID":"99ec7f91-11f2-43f8-93ce-04b319412bd2","Type":"ContainerStarted","Data":"6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d"}
Oct 30 01:23:47.083789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.083756    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" event={"ID":"99ec7f91-11f2-43f8-93ce-04b319412bd2","Type":"ContainerStarted","Data":"706df6737a1a82f8e863b946b10bf6d3dbb5fab9317be1a2aa54f4b0f4604f8c"}
Oct 30 01:23:47.087860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.087818    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr" event={"ID":"9a57814a-ee26-4472-862e-a0343b717010","Type":"ContainerStarted","Data":"6c21fce2eb0dd5b090a9861568414eb61587273b2414a06804822145f9ae7322"}
Oct 30 01:23:47.100479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.100418    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:47.122842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.122802    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:47.124493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.124322    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Oct 30 01:23:47.124493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.124381    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Oct 30 01:23:47.147228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.147145    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podStartSLOduration=2.147123439 podStartE2EDuration="2.147123439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:47.140997237 +0000 UTC m=+2468.688687734" watchObservedRunningTime="2026-10-30 01:23:47.147123439 +0000 UTC m=+2468.694814015"
Oct 30 01:23:47.449658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.449572    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr" podStartSLOduration=4.449553143 podStartE2EDuration="4.449553143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:47.216058128 +0000 UTC m=+2468.763748642" watchObservedRunningTime="2026-10-30 01:23:47.449553143 +0000 UTC m=+2468.997243640"
Oct 30 01:23:47.456286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.456245    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"]
Oct 30 01:23:47.486211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.486163    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:47.486521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.486490    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:47.488150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:47.486691    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:47.630794 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:47.630753    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:23:47.631678 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:47.631641    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:23:48.127114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:48.127028    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Oct 30 01:23:48.127114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:48.127092    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Oct 30 01:23:48.137614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:48.137570    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" event={"ID":"2c066f1b-c608-42af-800c-7167e4c48a80","Type":"ContainerStarted","Data":"826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f"}
Oct 30 01:23:48.137823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:48.137640    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" event={"ID":"2c066f1b-c608-42af-800c-7167e4c48a80","Type":"ContainerStarted","Data":"07906b3dd3c47892e3e3b3de86adfff22400358379d7b7569eb50dab1543e6a9"}
Oct 30 01:23:48.139788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:48.139756    5104 generic.go:356] "Generic (PLEG): container finished" podID="9a57814a-ee26-4472-862e-a0343b717010" containerID="6c21fce2eb0dd5b090a9861568414eb61587273b2414a06804822145f9ae7322" exitCode=0
Oct 30 01:23:48.139940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:48.139820    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr" event={"ID":"9a57814a-ee26-4472-862e-a0343b717010","Type":"ContainerDied","Data":"6c21fce2eb0dd5b090a9861568414eb61587273b2414a06804822145f9ae7322"}
Oct 30 01:23:48.219251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:48.219183    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podStartSLOduration=2.219160549 podStartE2EDuration="2.219160549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:48.182828875 +0000 UTC m=+2469.730519372" watchObservedRunningTime="2026-10-30 01:23:48.219160549 +0000 UTC m=+2469.766851046"
Oct 30 01:23:49.116546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.106199    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:49.116546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.106345    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Oct 30 01:23:49.116546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.106393    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Oct 30 01:23:49.130323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.130283    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Oct 30 01:23:49.130648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.130613    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Oct 30 01:23:49.742498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.737612    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:23:49.742498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.737706    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:49.845878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:49.845773    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:50.042392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.025087    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a57814a-ee26-4472-862e-a0343b717010-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a57814a-ee26-4472-862e-a0343b717010-maas-api-openshift-ca\") pod \"9a57814a-ee26-4472-862e-a0343b717010\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") "
Oct 30 01:23:50.042392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.025174    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a57814a-ee26-4472-862e-a0343b717010-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a57814a-ee26-4472-862e-a0343b717010-maas-api-xks-ca\") pod \"9a57814a-ee26-4472-862e-a0343b717010\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") "
Oct 30 01:23:50.042392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.025252    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a57814a-ee26-4472-862e-a0343b717010-kube-api-access-85zfn\" (UniqueName: \"kubernetes.io/projected/9a57814a-ee26-4472-862e-a0343b717010-kube-api-access-85zfn\") pod \"9a57814a-ee26-4472-862e-a0343b717010\" (UID: \"9a57814a-ee26-4472-862e-a0343b717010\") "
Oct 30 01:23:50.042392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.041168    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a57814a-ee26-4472-862e-a0343b717010-maas-api-openshift-ca" pod "9a57814a-ee26-4472-862e-a0343b717010" (UID: "9a57814a-ee26-4472-862e-a0343b717010"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:50.042392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.042276    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a57814a-ee26-4472-862e-a0343b717010-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.044557 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.044526    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a57814a-ee26-4472-862e-a0343b717010-kube-api-access-85zfn" pod "9a57814a-ee26-4472-862e-a0343b717010" (UID: "9a57814a-ee26-4472-862e-a0343b717010"). InnerVolumeSpecName "kube-api-access-85zfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:50.045809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.045780    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a57814a-ee26-4472-862e-a0343b717010-maas-api-xks-ca" pod "9a57814a-ee26-4472-862e-a0343b717010" (UID: "9a57814a-ee26-4472-862e-a0343b717010"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:50.147619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.146987    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Oct 30 01:23:50.147619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.147050    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Oct 30 01:23:50.147619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.147409    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Oct 30 01:23:50.147619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.147480    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Oct 30 01:23:50.148322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.148151    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a57814a-ee26-4472-862e-a0343b717010-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.148322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.148182    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85zfn\" (UniqueName: \"kubernetes.io/projected/9a57814a-ee26-4472-862e-a0343b717010-kube-api-access-85zfn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.200481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.200010    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"
Oct 30 01:23:50.234597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.234543    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr" event={"ID":"9a57814a-ee26-4472-862e-a0343b717010","Type":"ContainerDied","Data":"63ae4404cc06e9fe596ceb757092be2435f84531e3c7ce7865c3e66f6588ef37"}
Oct 30 01:23:50.234868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:50.234849    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ae4404cc06e9fe596ceb757092be2435f84531e3c7ce7865c3e66f6588ef37"
Oct 30 01:23:51.134528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:51.126438    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Oct 30 01:23:51.134528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:51.126519    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Oct 30 01:23:51.134528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:51.126961    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Oct 30 01:23:51.134528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:51.127001    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Oct 30 01:23:51.940704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:51.940671    5104 scope.go:122] "RemoveContainer" containerID="033b600d4763c483b96281b8a823f128fbfcc4cb2469f6f02375c15f3d81369e"
Oct 30 01:23:51.960818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:51.960793    5104 scope.go:122] "RemoveContainer" containerID="68dc0499127da8365dd540053111828fea9d30260ba38c134df06556916da0c9"
Oct 30 01:23:52.009793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.009636    5104 scope.go:122] "RemoveContainer" containerID="25ff49f286f26f2d63a0ec1c467ac563522255f807ab81770bcaf99d6d6054fe"
Oct 30 01:23:52.051361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.051333    5104 scope.go:122] "RemoveContainer" containerID="81f3f42c6aa050aedfeb0b51feefe2ea40295398cb5fd93d9791679e189a16dd"
Oct 30 01:23:52.063301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.063283    5104 scope.go:122] "RemoveContainer" containerID="f0d293708146bbd81a223903d984b9498f566e794165ebd038e54b7d9f2c928b"
Oct 30 01:23:52.107503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.104335    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Oct 30 01:23:52.107503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.104400    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Oct 30 01:23:52.120693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.120034    5104 scope.go:122] "RemoveContainer" containerID="48e7b0ef74b6207c8965d42724eb60d9d719d935dac0beedf63f00b48f08d249"
Oct 30 01:23:52.120693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.120574    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Oct 30 01:23:52.120693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.120630    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Oct 30 01:23:52.152652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:52.152628    5104 scope.go:122] "RemoveContainer" containerID="d9ddbe3e474f7ff6248139a37157fd3b501e77fa60290f9947b20eaf78a4d348"
Oct 30 01:23:53.123483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:53.123205    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Oct 30 01:23:53.123483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:53.123272    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Oct 30 01:23:53.146627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:53.146588    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Oct 30 01:23:53.148997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:53.147010    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Oct 30 01:23:54.100913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.100812    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Oct 30 01:23:54.100913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.100881    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Oct 30 01:23:54.125362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.125327    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:54.125831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.125817    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:54.126541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.126518    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:23:54.462206 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.462174    5104 generic.go:356] "Generic (PLEG): container finished" podID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerID="3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e" exitCode=0
Oct 30 01:23:54.462466 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.462204    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" event={"ID":"75cf53c2-ab95-4e5a-a404-4ed17a140a1c","Type":"ContainerDied","Data":"3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e"}
Oct 30 01:23:54.462902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.462883    5104 scope.go:122] "RemoveContainer" containerID="3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e"
Oct 30 01:23:54.465089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.464999    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:54.465089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.465099    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:54.466225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.465152    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:23:54.590762 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:54.588744    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache]"
Oct 30 01:23:54.882535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.873563    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"]
Oct 30 01:23:54.896477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.887258    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9a57814a-ee26-4472-862e-a0343b717010" containerName="revoke-keys"
Oct 30 01:23:54.896477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.887295    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a57814a-ee26-4472-862e-a0343b717010" containerName="revoke-keys"
Oct 30 01:23:54.896477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.887720    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a57814a-ee26-4472-862e-a0343b717010" containerName="revoke-keys"
Oct 30 01:23:54.909482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.905686    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"]
Oct 30 01:23:54.909482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.905816    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:54.938478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.929342    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:23:54.938478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.929545    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:23:54.938478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.929613    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:23:54.938478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:54.929656    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:55.008967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.008927    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsgz\" (UniqueName: \"kubernetes.io/projected/445248a7-9fa0-45c4-9278-579c204cc360-kube-api-access-tpsgz\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.009171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.009033    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.009171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.009090    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.101424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.101382    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Oct 30 01:23:55.101644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.101474    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Oct 30 01:23:55.110737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.110694    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsgz\" (UniqueName: \"kubernetes.io/projected/445248a7-9fa0-45c4-9278-579c204cc360-kube-api-access-tpsgz\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.110941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.110812    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.110941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.110878    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.111839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.111805    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.113407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.113382    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.129714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.129676    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsgz\" (UniqueName: \"kubernetes.io/projected/445248a7-9fa0-45c4-9278-579c204cc360-kube-api-access-tpsgz\") pod \"maas-api-revoke-sub-3fcd394d-vgbrn\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.243857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.243822    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:55.519538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.519422    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" event={"ID":"75cf53c2-ab95-4e5a-a404-4ed17a140a1c","Type":"ContainerStarted","Data":"cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4"}
Oct 30 01:23:55.588509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:55.584832    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"]
Oct 30 01:23:56.098641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.086593    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:23:56.098641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.098616    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 30 01:23:56.098934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.098682    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 30 01:23:56.107976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.107949    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:56.108393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.108372    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:56.109114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.109089    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:23:56.825584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.824753    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn" event={"ID":"445248a7-9fa0-45c4-9278-579c204cc360","Type":"ContainerStarted","Data":"8f27a8d8e066d69b5183afb38aa06727ead3d61484f8845d094f11e9ebad78c6"}
Oct 30 01:23:56.825584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.824895    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn" event={"ID":"445248a7-9fa0-45c4-9278-579c204cc360","Type":"ContainerStarted","Data":"0eef38b225eabd3d4debea1a6d5677ec3a9a1c43d71475ac964f4248216d5874"}
Oct 30 01:23:56.864399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:56.864341    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn" podStartSLOduration=2.8643266560000002 podStartE2EDuration="2.864326656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:56.863120359 +0000 UTC m=+2478.410810855" watchObservedRunningTime="2026-10-30 01:23:56.864326656 +0000 UTC m=+2478.412017152"
Oct 30 01:23:57.093538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:57.087567    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 30 01:23:57.093538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:57.087631    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 30 01:23:57.837131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:57.837092    5104 generic.go:356] "Generic (PLEG): container finished" podID="445248a7-9fa0-45c4-9278-579c204cc360" containerID="8f27a8d8e066d69b5183afb38aa06727ead3d61484f8845d094f11e9ebad78c6" exitCode=0
Oct 30 01:23:57.837648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:57.837176    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn" event={"ID":"445248a7-9fa0-45c4-9278-579c204cc360","Type":"ContainerDied","Data":"8f27a8d8e066d69b5183afb38aa06727ead3d61484f8845d094f11e9ebad78c6"}
Oct 30 01:23:58.092085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:58.091988    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 30 01:23:58.092431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:58.092395    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 30 01:23:58.956550 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:23:58.940575    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache]"
Oct 30 01:23:59.111485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.110652    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 30 01:23:59.111485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.110746    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 30 01:23:59.405506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.399643    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:23:59.536258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.536219    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-xks-ca\") pod \"445248a7-9fa0-45c4-9278-579c204cc360\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") "
Oct 30 01:23:59.536594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.536572    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-openshift-ca\") pod \"445248a7-9fa0-45c4-9278-579c204cc360\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") "
Oct 30 01:23:59.536852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.536832    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445248a7-9fa0-45c4-9278-579c204cc360-kube-api-access-tpsgz\" (UniqueName: \"kubernetes.io/projected/445248a7-9fa0-45c4-9278-579c204cc360-kube-api-access-tpsgz\") pod \"445248a7-9fa0-45c4-9278-579c204cc360\" (UID: \"445248a7-9fa0-45c4-9278-579c204cc360\") "
Oct 30 01:23:59.537017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.536980    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-openshift-ca" pod "445248a7-9fa0-45c4-9278-579c204cc360" (UID: "445248a7-9fa0-45c4-9278-579c204cc360"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:59.538186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.538163    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:59.538852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.538822    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-xks-ca" pod "445248a7-9fa0-45c4-9278-579c204cc360" (UID: "445248a7-9fa0-45c4-9278-579c204cc360"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:59.539154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.539135    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445248a7-9fa0-45c4-9278-579c204cc360-kube-api-access-tpsgz" pod "445248a7-9fa0-45c4-9278-579c204cc360" (UID: "445248a7-9fa0-45c4-9278-579c204cc360"). InnerVolumeSpecName "kube-api-access-tpsgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:59.640225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.640132    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpsgz\" (UniqueName: \"kubernetes.io/projected/445248a7-9fa0-45c4-9278-579c204cc360-kube-api-access-tpsgz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:23:59.640225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:23:59.640168    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445248a7-9fa0-45c4-9278-579c204cc360-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.154484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:00.090165    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 30 01:24:00.154484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:00.090240    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 30 01:24:00.239482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:00.238830    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn" event={"ID":"445248a7-9fa0-45c4-9278-579c204cc360","Type":"ContainerDied","Data":"0eef38b225eabd3d4debea1a6d5677ec3a9a1c43d71475ac964f4248216d5874"}
Oct 30 01:24:00.239482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:00.238898    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eef38b225eabd3d4debea1a6d5677ec3a9a1c43d71475ac964f4248216d5874"
Oct 30 01:24:00.239482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:00.238973    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"
Oct 30 01:24:01.110561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:01.103693    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 30 01:24:01.110561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:01.103807    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 30 01:24:02.118367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:02.118318    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:24:02.119545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:02.119512    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:24:02.120562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:02.120537    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:24:04.727542 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:04.727489    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:24:05.401483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.397650    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"]
Oct 30 01:24:05.401483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.399548    5104 cpu_manager.go:414] "containerMap: removing container" podUID="445248a7-9fa0-45c4-9278-579c204cc360" containerName="revoke-keys"
Oct 30 01:24:05.401483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.399577    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445248a7-9fa0-45c4-9278-579c204cc360" containerName="revoke-keys"
Oct 30 01:24:05.401483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.399884    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="445248a7-9fa0-45c4-9278-579c204cc360" containerName="revoke-keys"
Oct 30 01:24:05.431487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.425808    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.474400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.474357    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:24:05.474650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.474473    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"]
Oct 30 01:24:05.474982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.474956    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:24:05.475117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.475025    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:05.545736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.545700    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:24:05.549990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.549955    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.550181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.550086    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.550322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.550296    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jm45\" (UniqueName: \"kubernetes.io/projected/9c85f918-dfaa-41bf-96c3-d6a7095d1549-kube-api-access-8jm45\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.652690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.652528    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jm45\" (UniqueName: \"kubernetes.io/projected/9c85f918-dfaa-41bf-96c3-d6a7095d1549-kube-api-access-8jm45\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.652968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.652945    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.653072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.653053    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.653706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.653681    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.655763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.655738    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.671701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.671655    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jm45\" (UniqueName: \"kubernetes.io/projected/9c85f918-dfaa-41bf-96c3-d6a7095d1549-kube-api-access-8jm45\") pod \"maas-api-revoke-sub-d2cee3b1-7npzv\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:05.798406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:05.798371    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:06.126202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:06.126158    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"]
Oct 30 01:24:06.305974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:06.305926    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv" event={"ID":"9c85f918-dfaa-41bf-96c3-d6a7095d1549","Type":"ContainerStarted","Data":"29ce6999b0d6c2fd384c0e3eec6e7d7b6c21854a2f9c9ecf0b288beb724a7f56"}
Oct 30 01:24:07.364536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:07.361483    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv" event={"ID":"9c85f918-dfaa-41bf-96c3-d6a7095d1549","Type":"ContainerStarted","Data":"56e2d6d212decd59f06871d14b9793a031565b95c5b08b5f1ea5d5017dcfba0f"}
Oct 30 01:24:07.423467 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:07.423386    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv" podStartSLOduration=2.423368576 podStartE2EDuration="2.423368576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:07.417275065 +0000 UTC m=+2488.964965562" watchObservedRunningTime="2026-10-30 01:24:07.423368576 +0000 UTC m=+2488.971059095"
Oct 30 01:24:08.412529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:08.403727    5104 generic.go:356] "Generic (PLEG): container finished" podID="9c85f918-dfaa-41bf-96c3-d6a7095d1549" containerID="56e2d6d212decd59f06871d14b9793a031565b95c5b08b5f1ea5d5017dcfba0f" exitCode=0
Oct 30 01:24:08.412529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:08.403881    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv" event={"ID":"9c85f918-dfaa-41bf-96c3-d6a7095d1549","Type":"ContainerDied","Data":"56e2d6d212decd59f06871d14b9793a031565b95c5b08b5f1ea5d5017dcfba0f"}
Oct 30 01:24:09.107919 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:09.107883    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:24:09.627112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.627079    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:09.693634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.693602    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-openshift-ca\") pod \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") "
Oct 30 01:24:09.693869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.693725    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c85f918-dfaa-41bf-96c3-d6a7095d1549-kube-api-access-8jm45\" (UniqueName: \"kubernetes.io/projected/9c85f918-dfaa-41bf-96c3-d6a7095d1549-kube-api-access-8jm45\") pod \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") "
Oct 30 01:24:09.694236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.694183    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-openshift-ca" pod "9c85f918-dfaa-41bf-96c3-d6a7095d1549" (UID: "9c85f918-dfaa-41bf-96c3-d6a7095d1549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:09.694513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.694200    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-xks-ca\") pod \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\" (UID: \"9c85f918-dfaa-41bf-96c3-d6a7095d1549\") "
Oct 30 01:24:09.695952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.695922    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:09.696770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.696743    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c85f918-dfaa-41bf-96c3-d6a7095d1549-kube-api-access-8jm45" pod "9c85f918-dfaa-41bf-96c3-d6a7095d1549" (UID: "9c85f918-dfaa-41bf-96c3-d6a7095d1549"). InnerVolumeSpecName "kube-api-access-8jm45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:09.697541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.697513    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-xks-ca" pod "9c85f918-dfaa-41bf-96c3-d6a7095d1549" (UID: "9c85f918-dfaa-41bf-96c3-d6a7095d1549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:09.797472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.797431    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jm45\" (UniqueName: \"kubernetes.io/projected/9c85f918-dfaa-41bf-96c3-d6a7095d1549-kube-api-access-8jm45\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:09.797472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:09.797477    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c85f918-dfaa-41bf-96c3-d6a7095d1549-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.510339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:10.510300    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv" event={"ID":"9c85f918-dfaa-41bf-96c3-d6a7095d1549","Type":"ContainerDied","Data":"29ce6999b0d6c2fd384c0e3eec6e7d7b6c21854a2f9c9ecf0b288beb724a7f56"}
Oct 30 01:24:10.510637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:10.510617    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ce6999b0d6c2fd384c0e3eec6e7d7b6c21854a2f9c9ecf0b288beb724a7f56"
Oct 30 01:24:10.510803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:10.510791    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"
Oct 30 01:24:14.098577 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:14.098540    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache]"
Oct 30 01:24:14.750017 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:14.749975    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:24:24.873056 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:24.873010    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache]"
Oct 30 01:24:26.732477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:26.717585    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:24:27.060532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.060232    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:24:27.372046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.371941    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:24:27.495066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.437351    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:27.583029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.582955    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:27.650540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.631052    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:27.683131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.683073    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:27.720635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.720579    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:27.721570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.721543    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:24:27.731634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:27.731577    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:28.162940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.162898    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:28.208479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.205600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:24:28.208479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.206248    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:24:28.309718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.309669    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:28.559519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.554381    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:24:28.559519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.555305    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:24:28.559519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.555875    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:24:28.559519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.555940    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:24:28.609504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.593950    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:28.609504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.594521    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:24:28.703123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.703054    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:28.734660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.713615    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:28.805483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.778176    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 30 01:24:28.805483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:28.778521    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:29.215006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.214948    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:29.396619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.396557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:24:29.630479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.630347    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:24:29.646198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.646137    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:29.659480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.649497    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:29.659480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.659207    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:29.661916 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:29.661874    5104 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.052s"
Oct 30 01:24:29.662258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.662243    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:24:29.662443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.662424    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:24:29.662619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.662607    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:24:29.757842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.757760    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:24:29.759888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.759846    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:24:29.807565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.799038    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:29.868717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.868671    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:24:29.869065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:29.869009    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:30.393480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.375862    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:30.537561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.418000    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:24:30.560957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.560902    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:30.632541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.632442    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:30.676843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.676790    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:24:30.680104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.680072    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:24:30.731482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.710483    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:30.981532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.978569    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:24:30.981532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:30.980033    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:24:31.115654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.115602    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:24:31.390105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.390002    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:24:31.405832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.405771    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:31.501200 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:31.501147    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache]"
Oct 30 01:24:31.540615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.538470    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:31.571486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.566365    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 01:24:31.747169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.637560    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:31.850485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.678626    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:31.850485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.691411    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:31.850485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.750210    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:24:31.981735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.981680    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:31.989960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:31.989912    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:32.123635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.123116    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:24:32.126734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.126694    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:24:32.138952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.138568    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:24:32.209485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.188198    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:24:32.236901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.234892    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:24:32.265535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.261597    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:24:32.265535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.262908    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:24:32.265818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.265802    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:24:32.294965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.292326    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:24:32.312073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.296824    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:24:32.335178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.335129    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"]
Oct 30 01:24:32.361556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:32.355276    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"]
Oct 30 01:24:35.130515 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:35.130468    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice/crio-8a4813ab4f7312a988aec51b9a36447e43d889b54fe555a42e02c99ae9e4c10f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5555030_b09d_4df4_9632_7575b8d2f19f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:24:37.335612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.335551    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5" podUID="164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" containerName="payload-processing" containerID="cri-o://fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896" gracePeriod=25
Oct 30 01:24:37.356570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.356517    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z" podUID="a1fc686c-34eb-442b-80f1-623d1f780820" containerName="payload-pre-processing" containerID="cri-o://1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734" gracePeriod=25
Oct 30 01:24:37.725502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.725178    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:24:37.798709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.798603    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"]
Oct 30 01:24:37.800249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.800217    5104 cpu_manager.go:414] "containerMap: removing container" podUID="164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" containerName="payload-processing"
Oct 30 01:24:37.800249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.800249    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" containerName="payload-processing"
Oct 30 01:24:37.800507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.800359    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9c85f918-dfaa-41bf-96c3-d6a7095d1549" containerName="revoke-keys"
Oct 30 01:24:37.800507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.800368    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c85f918-dfaa-41bf-96c3-d6a7095d1549" containerName="revoke-keys"
Oct 30 01:24:37.800671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.800650    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c85f918-dfaa-41bf-96c3-d6a7095d1549" containerName="revoke-keys"
Oct 30 01:24:37.800749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.800680    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" containerName="payload-processing"
Oct 30 01:24:37.805870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.805835    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:24:37.806904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.806882    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:37.828522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.826721    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2a05c\""
Oct 30 01:24:37.828522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.827616    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2a05c-dockercfg-zltrz\""
Oct 30 01:24:37.837001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.834766    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"]
Oct 30 01:24:37.837224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.837171    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-plugins-config-volume\") pod \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") "
Oct 30 01:24:37.837336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.837319    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fc686c-34eb-442b-80f1-623d1f780820-kube-api-access-9s84m\" (UniqueName: \"kubernetes.io/projected/a1fc686c-34eb-442b-80f1-623d1f780820-kube-api-access-9s84m\") pod \"a1fc686c-34eb-442b-80f1-623d1f780820\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") "
Oct 30 01:24:37.837520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.837470    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1fc686c-34eb-442b-80f1-623d1f780820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fc686c-34eb-442b-80f1-623d1f780820-plugins-config-volume\") pod \"a1fc686c-34eb-442b-80f1-623d1f780820\" (UID: \"a1fc686c-34eb-442b-80f1-623d1f780820\") "
Oct 30 01:24:37.837599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.837525    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-kube-api-access-w4cmd\" (UniqueName: \"kubernetes.io/projected/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-kube-api-access-w4cmd\") pod \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\" (UID: \"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7\") "
Oct 30 01:24:37.838255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.838228    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58db24d1-9c08-43f2-86f1-af1d5c66b110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:37.838432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.838413    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fc686c-34eb-442b-80f1-623d1f780820-plugins-config-volume" pod "a1fc686c-34eb-442b-80f1-623d1f780820" (UID: "a1fc686c-34eb-442b-80f1-623d1f780820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:37.838644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.838615    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65s7\" (UniqueName: \"kubernetes.io/projected/58db24d1-9c08-43f2-86f1-af1d5c66b110-kube-api-access-v65s7\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:37.838795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.838757    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fc686c-34eb-442b-80f1-623d1f780820-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.838915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.838900    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-plugins-config-volume" pod "164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" (UID: "164a6d4e-d4c1-4a92-895a-2a0bf6a969e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:37.840930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.840893    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fc686c-34eb-442b-80f1-623d1f780820-kube-api-access-9s84m" pod "a1fc686c-34eb-442b-80f1-623d1f780820" (UID: "a1fc686c-34eb-442b-80f1-623d1f780820"). InnerVolumeSpecName "kube-api-access-9s84m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:37.845182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.845147    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"
Oct 30 01:24:37.845673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.845486    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5" event={"ID":"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7","Type":"ContainerDied","Data":"fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896"}
Oct 30 01:24:37.845673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.845537    5104 scope.go:122] "RemoveContainer" containerID="fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896"
Oct 30 01:24:37.854119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.848933    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-kube-api-access-w4cmd" pod "164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" (UID: "164a6d4e-d4c1-4a92-895a-2a0bf6a969e7"). InnerVolumeSpecName "kube-api-access-w4cmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:37.861682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.845445    5104 generic.go:356] "Generic (PLEG): container finished" podID="164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" containerID="fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896" exitCode=0
Oct 30 01:24:37.861982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.861953    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5" event={"ID":"164a6d4e-d4c1-4a92-895a-2a0bf6a969e7","Type":"ContainerDied","Data":"0db8ae7b559fdb3ea7b08fa8e38227cfc2c7a86aceb4c5258c96cd743b4642a5"}
Oct 30 01:24:37.865813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.865753    5104 generic.go:356] "Generic (PLEG): container finished" podID="a1fc686c-34eb-442b-80f1-623d1f780820" containerID="1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734" exitCode=0
Oct 30 01:24:37.865813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.865804    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"
Oct 30 01:24:37.866002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.865905    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z" event={"ID":"a1fc686c-34eb-442b-80f1-623d1f780820","Type":"ContainerDied","Data":"1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734"}
Oct 30 01:24:37.866002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.865953    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z" event={"ID":"a1fc686c-34eb-442b-80f1-623d1f780820","Type":"ContainerDied","Data":"98ea7869fca3daedf4dab42f0a7d7040eea1f7d8f108c8cac05b98e63b7cbe84"}
Oct 30 01:24:37.880049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.880026    5104 scope.go:122] "RemoveContainer" containerID="fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896"
Oct 30 01:24:37.880604 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:37.880572    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896\": container with ID starting with fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896 not found: ID does not exist" containerID="fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896"
Oct 30 01:24:37.880734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.880614    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896"} err="failed to get container status \"fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896\": rpc error: code = NotFound desc = could not find container \"fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896\": container with ID starting with fb164b56085053a73e3d135112a709b557fcf2c6aae8ed8709e624b721ce9896 not found: ID does not exist"
Oct 30 01:24:37.880734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.880638    5104 scope.go:122] "RemoveContainer" containerID="1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734"
Oct 30 01:24:37.975034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.948051    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65s7\" (UniqueName: \"kubernetes.io/projected/58db24d1-9c08-43f2-86f1-af1d5c66b110-kube-api-access-v65s7\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:37.975034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.948339    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58db24d1-9c08-43f2-86f1-af1d5c66b110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:37.975034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.948582    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.975034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.948603    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s84m\" (UniqueName: \"kubernetes.io/projected/a1fc686c-34eb-442b-80f1-623d1f780820-kube-api-access-9s84m\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.975034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:37.948620    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4cmd\" (UniqueName: \"kubernetes.io/projected/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7-kube-api-access-w4cmd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:24:38.045929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.023635    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65s7\" (UniqueName: \"kubernetes.io/projected/58db24d1-9c08-43f2-86f1-af1d5c66b110-kube-api-access-v65s7\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:38.047782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.046879    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58db24d1-9c08-43f2-86f1-af1d5c66b110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:38.050561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.050483    5104 scope.go:122] "RemoveContainer" containerID="1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734"
Oct 30 01:24:38.051036 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:38.050993    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734\": container with ID starting with 1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734 not found: ID does not exist" containerID="1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734"
Oct 30 01:24:38.051135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.051045    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734"} err="failed to get container status \"1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734\": rpc error: code = NotFound desc = could not find container \"1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734\": container with ID starting with 1ec5ea70fc78eee71462f4e7d162726bccf185c3e7e1e4d2b80c4246e67c8734 not found: ID does not exist"
Oct 30 01:24:38.080479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.080272    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"]
Oct 30 01:24:38.080479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.080383    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-b5fd7454c-4s82z"]
Oct 30 01:24:38.156207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.156169    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:38.200543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.200498    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"]
Oct 30 01:24:38.210101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.210047    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-668cfc5fcd-k7kp5"]
Oct 30 01:24:38.471615 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:24:38.471584    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58db24d1_9c08_43f2_86f1_af1d5c66b110.slice/crio-6d60835dc46bc596134f3d096eb0f1de3aa6c6611ba285770f39c87619decb08 WatchSource:0}: Error finding container 6d60835dc46bc596134f3d096eb0f1de3aa6c6611ba285770f39c87619decb08: Status 404 returned error can't find the container with id 6d60835dc46bc596134f3d096eb0f1de3aa6c6611ba285770f39c87619decb08
Oct 30 01:24:38.474558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.474529    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"]
Oct 30 01:24:38.585188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.585151    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164a6d4e-d4c1-4a92-895a-2a0bf6a969e7" path="/var/lib/kubelet/pods/164a6d4e-d4c1-4a92-895a-2a0bf6a969e7/volumes"
Oct 30 01:24:38.585892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.585866    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1fc686c-34eb-442b-80f1-623d1f780820" path="/var/lib/kubelet/pods/a1fc686c-34eb-442b-80f1-623d1f780820/volumes"
Oct 30 01:24:38.879640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.879597    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4" event={"ID":"58db24d1-9c08-43f2-86f1-af1d5c66b110","Type":"ContainerStarted","Data":"e89f46f12d281aa3285fd65b298bc4d12c010e58bf836fc37bd71b428f4d78f4"}
Oct 30 01:24:38.879945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.879921    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4" event={"ID":"58db24d1-9c08-43f2-86f1-af1d5c66b110","Type":"ContainerStarted","Data":"6d60835dc46bc596134f3d096eb0f1de3aa6c6611ba285770f39c87619decb08"}
Oct 30 01:24:38.880104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.880091    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:38.919133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:38.919071    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4" podStartSLOduration=18.919051554 podStartE2EDuration="18.919051554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:38.916647929 +0000 UTC m=+2520.464338426" watchObservedRunningTime="2026-10-30 01:24:38.919051554 +0000 UTC m=+2520.466742052"
Oct 30 01:24:39.159630 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:24:39.159546    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:24:42.949845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:42.949812    5104 generic.go:356] "Generic (PLEG): container finished" podID="1b131465-6995-4c54-909b-49205c47f220" containerID="ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1" exitCode=0
Oct 30 01:24:42.950297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:42.949886    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" event={"ID":"1b131465-6995-4c54-909b-49205c47f220","Type":"ContainerDied","Data":"ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1"}
Oct 30 01:24:42.950427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:42.950411    5104 scope.go:122] "RemoveContainer" containerID="ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1"
Oct 30 01:24:42.952640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:42.952610    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:24:42.952716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:42.952683    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:24:42.952754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:42.952740    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:24:43.965301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:43.965261    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" event={"ID":"1b131465-6995-4c54-909b-49205c47f220","Type":"ContainerStarted","Data":"fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028"}
Oct 30 01:24:44.794482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:44.793510    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:24:44.794482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:44.793567    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 01:24:44.794482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:44.793639    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 01:24:45.795481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:45.777384    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 01:24:45.795481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:45.777446    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 01:24:46.776993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:46.776953    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 01:24:46.777335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:46.777285    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 01:24:47.779605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:47.779555    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 01:24:47.780269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:47.780234    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 01:24:48.786481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:48.786354    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 30 01:24:48.786481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:48.786417    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 30 01:24:49.780246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.780207    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:24:49.780740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.780712    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:24:49.781479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.781440    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:24:49.880159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.880109    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"]
Oct 30 01:24:49.881764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.881731    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a1fc686c-34eb-442b-80f1-623d1f780820" containerName="payload-pre-processing"
Oct 30 01:24:49.886195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.886155    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1fc686c-34eb-442b-80f1-623d1f780820" containerName="payload-pre-processing"
Oct 30 01:24:49.886827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.886800    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1fc686c-34eb-442b-80f1-623d1f780820" containerName="payload-pre-processing"
Oct 30 01:24:49.898051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.898021    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:49.904917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.904737    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"]
Oct 30 01:24:49.905108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.905082    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:24:49.913172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.913147    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4be164\"/\"openshift-service-ca.crt\""
Oct 30 01:24:49.913875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.913854    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4be164\"/\"kube-root-ca.crt\""
Oct 30 01:24:49.914115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.914100    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4be164\"/\"default-dockercfg-xwbwp\""
Oct 30 01:24:49.914950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:49.914357    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4be164\"/\"ipp-route-4be164-kserve-self-signed-certs\""
Oct 30 01:24:50.029315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.029273    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-home\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.030332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.030239    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tls-certs\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.030659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.030637    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-dshm\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.031566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.031544    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tmp-dir\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.032228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.032206    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-model-cache\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.032393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.032375    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvpl\" (UniqueName: \"kubernetes.io/projected/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-kube-api-access-4nvpl\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.164480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.160635    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tmp-dir\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.164480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.160810    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-model-cache\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.164480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.160863    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvpl\" (UniqueName: \"kubernetes.io/projected/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-kube-api-access-4nvpl\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.164480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.161062    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-home\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.164480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.161209    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tls-certs\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.164480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.161289    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-dshm\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.164480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.162476    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tmp-dir\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.189479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.181908    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-model-cache\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.189479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.183366    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-home\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.296690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.292657    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-dshm\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.414991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.414196    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvpl\" (UniqueName: \"kubernetes.io/projected/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-kube-api-access-4nvpl\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.415941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.415587    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tls-certs\") pod \"ipp-route-4be164-kserve-85d9dfbd5-mtdd6\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") " pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:50.592228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:50.590892    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:24:51.022164 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:24:51.022120    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod753685cf_6a7e_4ecd_8fbf_a4173cd8dea5.slice/crio-e14291f14bd4b0dbba06c7728731b750d28975d97979e87bde9be253078348fe WatchSource:0}: Error finding container e14291f14bd4b0dbba06c7728731b750d28975d97979e87bde9be253078348fe: Status 404 returned error can't find the container with id e14291f14bd4b0dbba06c7728731b750d28975d97979e87bde9be253078348fe
Oct 30 01:24:51.035938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:51.033730    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"]
Oct 30 01:24:51.097673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:51.097631    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" event={"ID":"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5","Type":"ContainerStarted","Data":"e14291f14bd4b0dbba06c7728731b750d28975d97979e87bde9be253078348fe"}
Oct 30 01:24:52.119199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.119165    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" event={"ID":"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5","Type":"ContainerStarted","Data":"258c290869ead76d3bd1fc989d92418e0b25668f5d8eefa75cd04731da9540a2"}
Oct 30 01:24:52.156032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.155978    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podStartSLOduration=3.155963895 podStartE2EDuration="3.155963895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:52.155167571 +0000 UTC m=+2533.702858069" watchObservedRunningTime="2026-10-30 01:24:52.155963895 +0000 UTC m=+2533.703654420"
Oct 30 01:24:52.470204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.470166    5104 scope.go:122] "RemoveContainer" containerID="ff2db8057cd1ac2f6ccc1a112ae04b8e0e7bdc61cab7639cb1b20f76c661aa67"
Oct 30 01:24:52.485765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.485732    5104 scope.go:122] "RemoveContainer" containerID="9e597d44c4a5f0cd35df6d67276e22aa4216e9d750aeb171c6323f4ee46f71be"
Oct 30 01:24:52.505052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.504521    5104 scope.go:122] "RemoveContainer" containerID="0121ce474ef8a478a38df8b5cb855d81cfa5664100335021580e3df7b396b782"
Oct 30 01:24:52.521079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.521046    5104 scope.go:122] "RemoveContainer" containerID="cef0ba6f7c5e9020ee605b1f38b916c0bd41b3659e2ba91e9a21859839f7536a"
Oct 30 01:24:52.532892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.532807    5104 scope.go:122] "RemoveContainer" containerID="50416af5137c17b57fb0d3b27d33c68bc8f20a061e8820dc00051dc07312017a"
Oct 30 01:24:52.544704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.544677    5104 scope.go:122] "RemoveContainer" containerID="e7a17f517069198fe3edb78dc5ee1f04882f166297f11789fd1e8235b836f32d"
Oct 30 01:24:52.557140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.557113    5104 scope.go:122] "RemoveContainer" containerID="7d346b044524ab6db16e0ef2cbb5065a6f90963882a70c9db6cd60090181ddb1"
Oct 30 01:24:52.591709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:24:52.591683    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:25:00.592399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:00.592360    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:25:00.605692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:00.605660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:25:01.260691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:01.260660    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:25:04.936877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:04.936836    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"]
Oct 30 01:25:04.942202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:04.942181    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:04.962513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:04.962480    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"]
Oct 30 01:25:04.966380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:04.966349    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:25:04.966380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:04.966349    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:04.966621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:04.966590    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:25:04.966677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:04.966590    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:25:05.022475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.022424    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67dv\" (UniqueName: \"kubernetes.io/projected/c39b4e2c-273e-4005-871c-7fe990dd6b6c-kube-api-access-r67dv\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.022688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.022606    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.022846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.022825    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.124218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.124178    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67dv\" (UniqueName: \"kubernetes.io/projected/c39b4e2c-273e-4005-871c-7fe990dd6b6c-kube-api-access-r67dv\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.124426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.124270    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.124511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.124433    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.125125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.125095    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.126858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.126837    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.142362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.142338    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67dv\" (UniqueName: \"kubernetes.io/projected/c39b4e2c-273e-4005-871c-7fe990dd6b6c-kube-api-access-r67dv\") pod \"maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.260221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.260130    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:05.579136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:05.579100    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"]
Oct 30 01:25:05.579329 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:25:05.579303    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39b4e2c_273e_4005_871c_7fe990dd6b6c.slice/crio-d2385d92bcf563f96b2e4ec8d222b9251a84ef1b8d1b7f8b065531d47a76c845 WatchSource:0}: Error finding container d2385d92bcf563f96b2e4ec8d222b9251a84ef1b8d1b7f8b065531d47a76c845: Status 404 returned error can't find the container with id d2385d92bcf563f96b2e4ec8d222b9251a84ef1b8d1b7f8b065531d47a76c845
Oct 30 01:25:06.312355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:06.312245    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb" event={"ID":"c39b4e2c-273e-4005-871c-7fe990dd6b6c","Type":"ContainerStarted","Data":"b3d330eb91abce89ffbcae1ce08bdc319c3b7d4af4f23ae0155833e57825771b"}
Oct 30 01:25:06.312355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:06.312309    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb" event={"ID":"c39b4e2c-273e-4005-871c-7fe990dd6b6c","Type":"ContainerStarted","Data":"d2385d92bcf563f96b2e4ec8d222b9251a84ef1b8d1b7f8b065531d47a76c845"}
Oct 30 01:25:06.353972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:06.353915    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb" podStartSLOduration=2.353898801 podStartE2EDuration="2.353898801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:06.349931084 +0000 UTC m=+2547.897621581" watchObservedRunningTime="2026-10-30 01:25:06.353898801 +0000 UTC m=+2547.901589298"
Oct 30 01:25:07.324582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:07.324542    5104 generic.go:356] "Generic (PLEG): container finished" podID="c39b4e2c-273e-4005-871c-7fe990dd6b6c" containerID="b3d330eb91abce89ffbcae1ce08bdc319c3b7d4af4f23ae0155833e57825771b" exitCode=0
Oct 30 01:25:07.325000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:07.324603    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb" event={"ID":"c39b4e2c-273e-4005-871c-7fe990dd6b6c","Type":"ContainerDied","Data":"b3d330eb91abce89ffbcae1ce08bdc319c3b7d4af4f23ae0155833e57825771b"}
Oct 30 01:25:08.544324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.544291    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:08.705717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.705679    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39b4e2c-273e-4005-871c-7fe990dd6b6c-kube-api-access-r67dv\" (UniqueName: \"kubernetes.io/projected/c39b4e2c-273e-4005-871c-7fe990dd6b6c-kube-api-access-r67dv\") pod \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") "
Oct 30 01:25:08.705903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.705866    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-openshift-ca\") pod \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") "
Oct 30 01:25:08.706081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.706057    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-xks-ca\") pod \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\" (UID: \"c39b4e2c-273e-4005-871c-7fe990dd6b6c\") "
Oct 30 01:25:08.706348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.706313    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-openshift-ca" pod "c39b4e2c-273e-4005-871c-7fe990dd6b6c" (UID: "c39b4e2c-273e-4005-871c-7fe990dd6b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:08.708290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.708262    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.708905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.708876    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39b4e2c-273e-4005-871c-7fe990dd6b6c-kube-api-access-r67dv" pod "c39b4e2c-273e-4005-871c-7fe990dd6b6c" (UID: "c39b4e2c-273e-4005-871c-7fe990dd6b6c"). InnerVolumeSpecName "kube-api-access-r67dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:08.709278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.709258    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-xks-ca" pod "c39b4e2c-273e-4005-871c-7fe990dd6b6c" (UID: "c39b4e2c-273e-4005-871c-7fe990dd6b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:08.831508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.831359    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39b4e2c-273e-4005-871c-7fe990dd6b6c-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.831508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:08.831397    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r67dv\" (UniqueName: \"kubernetes.io/projected/c39b4e2c-273e-4005-871c-7fe990dd6b6c-kube-api-access-r67dv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:09.107176 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:25:09.107071    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:25:09.344960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:09.344927    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"
Oct 30 01:25:09.344960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:09.344953    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb" event={"ID":"c39b4e2c-273e-4005-871c-7fe990dd6b6c","Type":"ContainerDied","Data":"d2385d92bcf563f96b2e4ec8d222b9251a84ef1b8d1b7f8b065531d47a76c845"}
Oct 30 01:25:09.345181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:09.345000    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2385d92bcf563f96b2e4ec8d222b9251a84ef1b8d1b7f8b065531d47a76c845"
Oct 30 01:25:13.461078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:13.461042    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:25:15.640663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:15.640611    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:15.683492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:15.673632    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:15.889404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:15.889346    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:25:15.891956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:15.891871    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:15.907341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:15.907247    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:15.923475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:15.917885    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:16.284664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:16.284604    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:16.313793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:16.313737    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:16.465271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:16.463777    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:16.717480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:16.712551    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:25:16.717480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:16.713165    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:25:16.894501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:16.892989    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:16.894501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:16.893365    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:25:17.335418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.263331    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:17.335418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.320865    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:17.335418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.330444    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:25:17.496964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.398937    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:25:17.683476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.585764    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:17.708491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.694623    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"]
Oct 30 01:25:17.708491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.694747    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:25:17.708491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.648562    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:17.757839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.727957    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:17.757839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.585807    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:17.791480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.775395    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:25:17.880634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.874334    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9" podUID="96d0232a-84d5-44ff-b2b6-d45f390c392d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:18.021184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.898519    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d0a7b-ssx5j"]
Oct 30 01:25:18.021184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.898607    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:25:18.021184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:17.926086    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:18.079309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.079275    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:25:18.105823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.105209    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:25:18.294592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.268562    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:18.294592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.272690    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:25:18.294592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.285796    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:18.327224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.327175    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:18.620662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.619305    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be997695-27a5-4279-96a0-35cac451f1f8" path="/var/lib/kubelet/pods/be997695-27a5-4279-96a0-35cac451f1f8/volumes"
Oct 30 01:25:18.677978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.677943    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:25:18.701272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.679043    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:25:18.702132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.702102    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:25:18.711964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.711916    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:18.732577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.729589    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:18.732577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.729877    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:25:18.732862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.732823    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:25:18.732926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.732891    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:25:18.757844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.757817    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:25:18.759899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.759853    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:25:18.760357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.760335    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:25:18.764293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.764033    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:25:18.819606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:18.819574    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:25:19.110515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:19.099884    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"]
Oct 30 01:25:19.122517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:19.122486    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f85cbc6-jlcx8"]
Oct 30 01:25:20.713500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:20.713446    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee055e72-2e4b-4e4f-82c4-9649c6dffb11" path="/var/lib/kubelet/pods/ee055e72-2e4b-4e4f-82c4-9649c6dffb11/volumes"
Oct 30 01:25:27.032400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:27.032357    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"]
Oct 30 01:25:27.039817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:27.039786    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f57134-vz7ns"]
Oct 30 01:25:28.593408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:28.593322    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27cb232f-7fad-42f8-8219-cbf56755b74f" path="/var/lib/kubelet/pods/27cb232f-7fad-42f8-8219-cbf56755b74f/volumes"
Oct 30 01:25:29.036611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.036575    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"]
Oct 30 01:25:29.047985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.047948    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba142b9-pj586"]
Oct 30 01:25:29.213732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.213697    5104 generic.go:356] "Generic (PLEG): container finished" podID="bd41d294-abf2-4b68-bec1-f29971fca466" containerID="b2002898df8c659365fc2e8744c05f4c17b9065db867f550e7a4a3826d6a7544" exitCode=0
Oct 30 01:25:29.213958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.213772    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" event={"ID":"bd41d294-abf2-4b68-bec1-f29971fca466","Type":"ContainerDied","Data":"b2002898df8c659365fc2e8744c05f4c17b9065db867f550e7a4a3826d6a7544"}
Oct 30 01:25:29.214376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.214353    5104 scope.go:122] "RemoveContainer" containerID="b2002898df8c659365fc2e8744c05f4c17b9065db867f550e7a4a3826d6a7544"
Oct 30 01:25:29.216749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.216717    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:29.216844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.216793    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:29.216844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:29.216827    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:30.227832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:30.227787    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" event={"ID":"bd41d294-abf2-4b68-bec1-f29971fca466","Type":"ContainerStarted","Data":"9d1ba47679fd8dfe61396c0aeded0563b7db0d82ba3db2e437c6017fc85ce242"}
Oct 30 01:25:30.420541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:30.420503    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:25:30.420765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:30.420692    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 30 01:25:30.420765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:30.420735    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 30 01:25:30.600603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:30.598401    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9510bd79-7dd9-414a-ae0d-ea6d775a1970" path="/var/lib/kubelet/pods/9510bd79-7dd9-414a-ae0d-ea6d775a1970/volumes"
Oct 30 01:25:31.424168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:31.424123    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 30 01:25:31.429412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:31.428631    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 30 01:25:32.420261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:32.420208    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 30 01:25:32.420487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:32.420287    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 30 01:25:33.420826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:33.420780    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 30 01:25:33.421371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:33.420855    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 30 01:25:34.420361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:34.420320    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 30 01:25:34.420758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:34.420380    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 30 01:25:35.420694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:35.420655    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 30 01:25:35.421134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:35.420716    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 30 01:25:36.438976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:36.438944    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:25:36.439419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:36.439393    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:25:36.440235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:36.440204    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:25:38.806534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:38.806496    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"]
Oct 30 01:25:38.807328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:38.807267    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw" podUID="45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" containerName="payload-pre-processing" containerID="cri-o://5df04ae08139cc1207d9c2c2959ca0c69e6fb4894313f1f96729c24e368e1317" gracePeriod=30
Oct 30 01:25:39.651534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.651494    5104 generic.go:356] "Generic (PLEG): container finished" podID="45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" containerID="5df04ae08139cc1207d9c2c2959ca0c69e6fb4894313f1f96729c24e368e1317" exitCode=0
Oct 30 01:25:39.652208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.652176    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw" event={"ID":"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9","Type":"ContainerDied","Data":"5df04ae08139cc1207d9c2c2959ca0c69e6fb4894313f1f96729c24e368e1317"}
Oct 30 01:25:39.738180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.721363    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"]
Oct 30 01:25:39.743372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.742479    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d32079-bxfxb"]
Oct 30 01:25:39.753559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.753269    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:25:39.781495 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:25:39.781064    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:25:39.845157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.839126    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-plugins-config-volume\") pod \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") "
Oct 30 01:25:39.845157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.839611    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-kube-api-access-bq7vj\" (UniqueName: \"kubernetes.io/projected/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-kube-api-access-bq7vj\") pod \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\" (UID: \"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9\") "
Oct 30 01:25:39.845157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.843912    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-plugins-config-volume" pod "45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" (UID: "45040b2d-a8b3-4d6e-95a0-f450aeae6ec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:39.903731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.902144    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-kube-api-access-bq7vj" pod "45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" (UID: "45040b2d-a8b3-4d6e-95a0-f450aeae6ec9"). InnerVolumeSpecName "kube-api-access-bq7vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:39.943505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.943434    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq7vj\" (UniqueName: \"kubernetes.io/projected/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-kube-api-access-bq7vj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:39.943505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:39.943482    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:40.035267 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.035226    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"]
Oct 30 01:25:40.041408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.041370    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7ff1cd-mzzjv"]
Oct 30 01:25:40.584092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.584059    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19921e74-8ff6-44ec-ad17-cdd9207d9848" path="/var/lib/kubelet/pods/19921e74-8ff6-44ec-ad17-cdd9207d9848/volumes"
Oct 30 01:25:40.584707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.584688    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39b4e2c-273e-4005-871c-7fe990dd6b6c" path="/var/lib/kubelet/pods/c39b4e2c-273e-4005-871c-7fe990dd6b6c/volumes"
Oct 30 01:25:40.665707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.665673    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"
Oct 30 01:25:40.665707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.665693    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw" event={"ID":"45040b2d-a8b3-4d6e-95a0-f450aeae6ec9","Type":"ContainerDied","Data":"08edb3ab2840ba77e3bcc011722cdbb7926c59bdebaeb1ab248d12e86b29d230"}
Oct 30 01:25:40.665973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.665765    5104 scope.go:122] "RemoveContainer" containerID="5df04ae08139cc1207d9c2c2959ca0c69e6fb4894313f1f96729c24e368e1317"
Oct 30 01:25:40.736370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.736328    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"]
Oct 30 01:25:40.736627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:40.736406    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d32079-86d7564b5f-wvddw"]
Oct 30 01:25:42.582326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.582285    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" path="/var/lib/kubelet/pods/45040b2d-a8b3-4d6e-95a0-f450aeae6ec9/volumes"
Oct 30 01:25:42.777497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.777442    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"]
Oct 30 01:25:42.779339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.779303    5104 cpu_manager.go:414] "containerMap: removing container" podUID="45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" containerName="payload-pre-processing"
Oct 30 01:25:42.779339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.779337    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" containerName="payload-pre-processing"
Oct 30 01:25:42.779655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.779420    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c39b4e2c-273e-4005-871c-7fe990dd6b6c" containerName="revoke-keys"
Oct 30 01:25:42.779655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.779428    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39b4e2c-273e-4005-871c-7fe990dd6b6c" containerName="revoke-keys"
Oct 30 01:25:42.779742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.779716    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39b4e2c-273e-4005-871c-7fe990dd6b6c" containerName="revoke-keys"
Oct 30 01:25:42.779775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.779743    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="45040b2d-a8b3-4d6e-95a0-f450aeae6ec9" containerName="payload-pre-processing"
Oct 30 01:25:42.786492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.786451    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:42.797852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.797816    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"]
Oct 30 01:25:42.800731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.800701    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-368bbb\""
Oct 30 01:25:42.840987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.840890    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"]
Oct 30 01:25:42.846826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.846794    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:42.865079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.864670    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"]
Oct 30 01:25:42.865079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.865020    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-368bbb-dockercfg-d8ddd\""
Oct 30 01:25:42.946488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.928343    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:42.946488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.928478    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqs24\" (UniqueName: \"kubernetes.io/projected/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-kube-api-access-lqs24\") pod \"payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:42.946488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.928559    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwwv\" (UniqueName: \"kubernetes.io/projected/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-kube-api-access-kbwwv\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:42.946488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:42.928845    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:43.116262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.033936    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:43.156482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.153251    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:43.156482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.153385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqs24\" (UniqueName: \"kubernetes.io/projected/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-kube-api-access-lqs24\") pod \"payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:43.156482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.153495    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwwv\" (UniqueName: \"kubernetes.io/projected/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-kube-api-access-kbwwv\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:43.156482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.064623    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:43.299479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.294979    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:43.359480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.353246    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwwv\" (UniqueName: \"kubernetes.io/projected/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-kube-api-access-kbwwv\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:43.359480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.354658    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqs24\" (UniqueName: \"kubernetes.io/projected/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-kube-api-access-lqs24\") pod \"payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:43.437483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.432406    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:43.472354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.472047    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:43.919642 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:25:43.919597    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e05841_5a7e_42f7_a1c8_7bda50c33cdd.slice/crio-20c991adbdfc01820b8e76a33be91cbf3b681f81909c69cabf756e83666e45e0 WatchSource:0}: Error finding container 20c991adbdfc01820b8e76a33be91cbf3b681f81909c69cabf756e83666e45e0: Status 404 returned error can't find the container with id 20c991adbdfc01820b8e76a33be91cbf3b681f81909c69cabf756e83666e45e0
Oct 30 01:25:43.921292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.921258    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"]
Oct 30 01:25:43.928940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:43.928910    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"]
Oct 30 01:25:43.929714 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:25:43.929578    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a51238_3b16_4e4a_b87b_a19ddf8af5dd.slice/crio-7b6389b5d59dedc0f41194e7101859c734d3d7b29f500d7ec27c3da66db885a8 WatchSource:0}: Error finding container 7b6389b5d59dedc0f41194e7101859c734d3d7b29f500d7ec27c3da66db885a8: Status 404 returned error can't find the container with id 7b6389b5d59dedc0f41194e7101859c734d3d7b29f500d7ec27c3da66db885a8
Oct 30 01:25:44.744550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.744491    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt" event={"ID":"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd","Type":"ContainerStarted","Data":"b1d573d4771f1dd25cae4641a9fdec70bd57787ac2dcb63957af72fca1637fbd"}
Oct 30 01:25:44.744799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.744562    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt" event={"ID":"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd","Type":"ContainerStarted","Data":"7b6389b5d59dedc0f41194e7101859c734d3d7b29f500d7ec27c3da66db885a8"}
Oct 30 01:25:44.744799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.744594    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:44.747091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.747060    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh" event={"ID":"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd","Type":"ContainerStarted","Data":"ac69a29c010e5b03b1c84176a7591f3e10f7e1b6e5b82583ae0aaba3ae0d42df"}
Oct 30 01:25:44.747252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.747116    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh" event={"ID":"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd","Type":"ContainerStarted","Data":"20c991adbdfc01820b8e76a33be91cbf3b681f81909c69cabf756e83666e45e0"}
Oct 30 01:25:44.747252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.747147    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:44.789944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.785909    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt" podStartSLOduration=2.785889148 podStartE2EDuration="2.785889148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:44.783239671 +0000 UTC m=+2586.330930167" watchObservedRunningTime="2026-10-30 01:25:44.785889148 +0000 UTC m=+2586.333579647"
Oct 30 01:25:44.830479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:44.830115    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh" podStartSLOduration=2.83009453 podStartE2EDuration="2.83009453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:44.822855252 +0000 UTC m=+2586.370545749" watchObservedRunningTime="2026-10-30 01:25:44.83009453 +0000 UTC m=+2586.377785029"
Oct 30 01:25:46.170586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.170552    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"]
Oct 30 01:25:46.171408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.171372    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy" containerID="cri-o://f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8" gracePeriod=30
Oct 30 01:25:46.861687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.854448    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"]
Oct 30 01:25:46.959659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.953794    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.009218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.986410    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:47.009218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.986626    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:25:47.009218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.986711    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:25:47.009218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:46.986419    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:25:47.075719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.028694    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.075719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.028763    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlftz\" (UniqueName: \"kubernetes.io/projected/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-kube-api-access-wlftz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.075719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.028951    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.075719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.052519    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"]
Oct 30 01:25:47.152593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.152073    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.152593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.152298    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.152593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.152331    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlftz\" (UniqueName: \"kubernetes.io/projected/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-kube-api-access-wlftz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.153302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.153277    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.182140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.182105    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.184808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.184773    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlftz\" (UniqueName: \"kubernetes.io/projected/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-kube-api-access-wlftz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.416276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.415821    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:47.769861 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.769823    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"]
Oct 30 01:25:47.775663 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:25:47.775632    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbdaf72_d41c_42d3_8cf9_8a6c898a8ecf.slice/crio-84e8a439d3c874b8625bcbc45847f83f880566e6658377d0aca5c88f9e06bf07 WatchSource:0}: Error finding container 84e8a439d3c874b8625bcbc45847f83f880566e6658377d0aca5c88f9e06bf07: Status 404 returned error can't find the container with id 84e8a439d3c874b8625bcbc45847f83f880566e6658377d0aca5c88f9e06bf07
Oct 30 01:25:47.999656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:47.999612    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k" event={"ID":"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf","Type":"ContainerStarted","Data":"84e8a439d3c874b8625bcbc45847f83f880566e6658377d0aca5c88f9e06bf07"}
Oct 30 01:25:49.000938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.000900    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"]
Oct 30 01:25:49.007790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.007758    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.011982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.011952    5104 generic.go:356] "Generic (PLEG): container finished" podID="afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" containerID="05966e0160a5291b5f6ee06e181b82f1d950fff6b37389496ed0911652b967e0" exitCode=0
Oct 30 01:25:49.012133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.012038    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k" event={"ID":"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf","Type":"ContainerDied","Data":"05966e0160a5291b5f6ee06e181b82f1d950fff6b37389496ed0911652b967e0"}
Oct 30 01:25:49.023249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.023218    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"]
Oct 30 01:25:49.117950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.117907    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.118165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.118061    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.118274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.118256    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85dk\" (UniqueName: \"kubernetes.io/projected/3370d60f-e20d-4420-b2cb-32d15d5e3e59-kube-api-access-h85dk\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.220985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.220937    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85dk\" (UniqueName: \"kubernetes.io/projected/3370d60f-e20d-4420-b2cb-32d15d5e3e59-kube-api-access-h85dk\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.221226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.221200    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.221311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.221292    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.221922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.221900    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.223817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.223793    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.239274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.239239    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85dk\" (UniqueName: \"kubernetes.io/projected/3370d60f-e20d-4420-b2cb-32d15d5e3e59-kube-api-access-h85dk\") pod \"maas-api-revoke-sub-7db92d77-fnbv7\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.331205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.331168    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:49.644176 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:25:49.644145    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3370d60f_e20d_4420_b2cb_32d15d5e3e59.slice/crio-35ba8de2c93a2bf446c06a1ea791ab1bdf753555c1201ce3eb78396e7f569ecc WatchSource:0}: Error finding container 35ba8de2c93a2bf446c06a1ea791ab1bdf753555c1201ce3eb78396e7f569ecc: Status 404 returned error can't find the container with id 35ba8de2c93a2bf446c06a1ea791ab1bdf753555c1201ce3eb78396e7f569ecc
Oct 30 01:25:49.645086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:49.645056    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"]
Oct 30 01:25:50.024390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.024348    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7" event={"ID":"3370d60f-e20d-4420-b2cb-32d15d5e3e59","Type":"ContainerStarted","Data":"35ba8de2c93a2bf446c06a1ea791ab1bdf753555c1201ce3eb78396e7f569ecc"}
Oct 30 01:25:50.280131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.242692    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:50.456659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.456627    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-kube-api-access-wlftz\" (UniqueName: \"kubernetes.io/projected/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-kube-api-access-wlftz\") pod \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") "
Oct 30 01:25:50.456821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.456771    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-xks-ca\") pod \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") "
Oct 30 01:25:50.457077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.457061    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-openshift-ca\") pod \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\" (UID: \"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf\") "
Oct 30 01:25:50.457561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.457525    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-openshift-ca" pod "afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" (UID: "afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:50.458039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.458015    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:50.458932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.458908    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-kube-api-access-wlftz" pod "afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" (UID: "afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf"). InnerVolumeSpecName "kube-api-access-wlftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:50.459024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.458936    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-xks-ca" pod "afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" (UID: "afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:50.559427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.559333    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlftz\" (UniqueName: \"kubernetes.io/projected/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-kube-api-access-wlftz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:50.559427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:50.559375    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.035966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.035926    5104 generic.go:356] "Generic (PLEG): container finished" podID="3370d60f-e20d-4420-b2cb-32d15d5e3e59" containerID="fcc571f8f8f2fbde62261c7574a62d05ccff07344bc3349087cfe7aaeeeda1a9" exitCode=0
Oct 30 01:25:51.036425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.036015    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7" event={"ID":"3370d60f-e20d-4420-b2cb-32d15d5e3e59","Type":"ContainerDied","Data":"fcc571f8f8f2fbde62261c7574a62d05ccff07344bc3349087cfe7aaeeeda1a9"}
Oct 30 01:25:51.037894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.037868    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"
Oct 30 01:25:51.038013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.037874    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k" event={"ID":"afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf","Type":"ContainerDied","Data":"84e8a439d3c874b8625bcbc45847f83f880566e6658377d0aca5c88f9e06bf07"}
Oct 30 01:25:51.038062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.038010    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84e8a439d3c874b8625bcbc45847f83f880566e6658377d0aca5c88f9e06bf07"
Oct 30 01:25:51.505803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.505766    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:25:51.561157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.561123    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"]
Oct 30 01:25:51.562098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.562081    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy"
Oct 30 01:25:51.562155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.562100    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy"
Oct 30 01:25:51.562200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.562165    5104 cpu_manager.go:414] "containerMap: removing container" podUID="afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" containerName="revoke-keys"
Oct 30 01:25:51.562200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.562171    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" containerName="revoke-keys"
Oct 30 01:25:51.562341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.562331    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy"
Oct 30 01:25:51.562380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.562355    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" containerName="revoke-keys"
Oct 30 01:25:51.566592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.566565    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.588448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.587906    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-envoy\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.588448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.588004    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-data\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.588448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.588343    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-data" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:51.589077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.589057    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-istio-token\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-istio-token\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.589171    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-socket\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.589495    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-socket" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.589577    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-kube-api-access-c6jmb\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-kube-api-access-c6jmb\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.590086    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c888e628-a915-4c08-b192-10976b6148cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c888e628-a915-4c08-b192-10976b6148cc-istiod-ca-cert\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.590447    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c888e628-a915-4c08-b192-10976b6148cc-istiod-ca-cert" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.590641    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-credential-socket\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.590935    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-credential-socket" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.591000    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c888e628-a915-4c08-b192-10976b6148cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c888e628-a915-4c08-b192-10976b6148cc-istio-podinfo\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.591433    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-certs\") pod \"c888e628-a915-4c08-b192-10976b6148cc\" (UID: \"c888e628-a915-4c08-b192-10976b6148cc\") "
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.591921    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.592019    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.592250    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.592658    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39713818-355a-4c98-b1b0-9c7da92a6803-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.592713    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.592751    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593152    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lphct\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-kube-api-access-lphct\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593233    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593395    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39713818-355a-4c98-b1b0-9c7da92a6803-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593608    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593630    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593643    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c888e628-a915-4c08-b192-10976b6148cc-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593676    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.593948    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-certs" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:51.598675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.596755    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"]
Oct 30 01:25:51.604543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.604515    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-istio-token" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:51.615977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.615937    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-envoy" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:51.621834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.621786    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c888e628-a915-4c08-b192-10976b6148cc-istio-podinfo" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:25:51.622066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.621856    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-kube-api-access-c6jmb" pod "c888e628-a915-4c08-b192-10976b6148cc" (UID: "c888e628-a915-4c08-b192-10976b6148cc"). InnerVolumeSpecName "kube-api-access-c6jmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:51.695250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695210    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.695504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695484    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39713818-355a-4c98-b1b0-9c7da92a6803-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.695615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695520    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.695709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695664    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.695776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695713    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lphct\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-kube-api-access-lphct\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.695776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695756    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.695902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695858    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39713818-355a-4c98-b1b0-9c7da92a6803-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.695902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695894    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.696016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.695933    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.696096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696070    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.696167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696079    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c888e628-a915-4c08-b192-10976b6148cc-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.696167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696128    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.696167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696145    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c888e628-a915-4c08-b192-10976b6148cc-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.696167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696159    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.696367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696174    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6jmb\" (UniqueName: \"kubernetes.io/projected/c888e628-a915-4c08-b192-10976b6148cc-kube-api-access-c6jmb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:51.696367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696192    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39713818-355a-4c98-b1b0-9c7da92a6803-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.696367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696317    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.696587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696509    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.696642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.696605    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.698655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.698631    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.699352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.699334    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39713818-355a-4c98-b1b0-9c7da92a6803-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.713650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.713578    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.714032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.714007    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lphct\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-kube-api-access-lphct\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:51.932628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:51.916920    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:52.056868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.056829    5104 generic.go:356] "Generic (PLEG): container finished" podID="c888e628-a915-4c08-b192-10976b6148cc" containerID="f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8" exitCode=0
Oct 30 01:25:52.057240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.057139    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"
Oct 30 01:25:52.061813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.061773    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" event={"ID":"c888e628-a915-4c08-b192-10976b6148cc","Type":"ContainerDied","Data":"f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8"}
Oct 30 01:25:52.061961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.061848    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp" event={"ID":"c888e628-a915-4c08-b192-10976b6148cc","Type":"ContainerDied","Data":"9556c90a154c798882a7beeee0ed1b447a4dd0f95940dda157914fd2b55564ea"}
Oct 30 01:25:52.061961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.061883    5104 scope.go:122] "RemoveContainer" containerID="f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8"
Oct 30 01:25:52.085067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.083821    5104 scope.go:122] "RemoveContainer" containerID="f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5"
Oct 30 01:25:52.099796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.099771    5104 scope.go:122] "RemoveContainer" containerID="f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8"
Oct 30 01:25:52.100373 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:25:52.100328    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8\": container with ID starting with f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8 not found: ID does not exist" containerID="f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8"
Oct 30 01:25:52.100505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.100388    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8"} err="failed to get container status \"f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8\": rpc error: code = NotFound desc = could not find container \"f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8\": container with ID starting with f294bc0daa9e37cb14bc70535816bcb31b0b7361b63c958e66c968de824872c8 not found: ID does not exist"
Oct 30 01:25:52.100505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.100418    5104 scope.go:122] "RemoveContainer" containerID="f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5"
Oct 30 01:25:52.100884 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:25:52.100860    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5\": container with ID starting with f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5 not found: ID does not exist" containerID="f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5"
Oct 30 01:25:52.100949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.100895    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5"} err="failed to get container status \"f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5\": rpc error: code = NotFound desc = could not find container \"f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5\": container with ID starting with f2c5957cf99bd8b926d628f04a3657b7673219fe84a3d6c6ab268c45e99b01d5 not found: ID does not exist"
Oct 30 01:25:52.120436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.118824    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"]
Oct 30 01:25:52.127916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.127881    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d32079-openshift-default-7b55d86997-4cjvp"]
Oct 30 01:25:52.288587 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:25:52.288553    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39713818_355a_4c98_b1b0_9c7da92a6803.slice/crio-cd50deb74f9288828318e23e7603af4628d1a05ac180c258dc30972daa06ac00 WatchSource:0}: Error finding container cd50deb74f9288828318e23e7603af4628d1a05ac180c258dc30972daa06ac00: Status 404 returned error can't find the container with id cd50deb74f9288828318e23e7603af4628d1a05ac180c258dc30972daa06ac00
Oct 30 01:25:52.289195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.289179    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:52.290089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.290067    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"]
Oct 30 01:25:52.292335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.292294    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:52.292484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.292369    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:52.292484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.292398    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:25:52.430534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.430489    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-xks-ca\") pod \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") "
Oct 30 01:25:52.431006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.430980    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-openshift-ca\") pod \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") "
Oct 30 01:25:52.431159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.431106    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3370d60f-e20d-4420-b2cb-32d15d5e3e59-kube-api-access-h85dk\" (UniqueName: \"kubernetes.io/projected/3370d60f-e20d-4420-b2cb-32d15d5e3e59-kube-api-access-h85dk\") pod \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\" (UID: \"3370d60f-e20d-4420-b2cb-32d15d5e3e59\") "
Oct 30 01:25:52.431574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.431543    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-openshift-ca" pod "3370d60f-e20d-4420-b2cb-32d15d5e3e59" (UID: "3370d60f-e20d-4420-b2cb-32d15d5e3e59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:52.432102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.432075    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:52.433115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.433086    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-xks-ca" pod "3370d60f-e20d-4420-b2cb-32d15d5e3e59" (UID: "3370d60f-e20d-4420-b2cb-32d15d5e3e59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:52.433671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.433650    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3370d60f-e20d-4420-b2cb-32d15d5e3e59-kube-api-access-h85dk" pod "3370d60f-e20d-4420-b2cb-32d15d5e3e59" (UID: "3370d60f-e20d-4420-b2cb-32d15d5e3e59"). InnerVolumeSpecName "kube-api-access-h85dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:52.534755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.534565    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h85dk\" (UniqueName: \"kubernetes.io/projected/3370d60f-e20d-4420-b2cb-32d15d5e3e59-kube-api-access-h85dk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:52.534755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.534593    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3370d60f-e20d-4420-b2cb-32d15d5e3e59-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:52.586598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.586556    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c888e628-a915-4c08-b192-10976b6148cc" path="/var/lib/kubelet/pods/c888e628-a915-4c08-b192-10976b6148cc/volumes"
Oct 30 01:25:52.658783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.658748    5104 scope.go:122] "RemoveContainer" containerID="a54479e95cb2b55785b75a1cfdb017820ea73fe0095532c66d326a148719481c"
Oct 30 01:25:52.671913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.671889    5104 scope.go:122] "RemoveContainer" containerID="2e47d2c9547695cc2c8da84f2da86e8f41b6ab601e54609261acbf776e98d90f"
Oct 30 01:25:52.687198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:52.687172    5104 scope.go:122] "RemoveContainer" containerID="972d54775caaae2de591865a0af937cd9d67fe8d56a1b69fb2831d8e3a7e9cde"
Oct 30 01:25:53.092483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.089176    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" event={"ID":"39713818-355a-4c98-b1b0-9c7da92a6803","Type":"ContainerStarted","Data":"57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e"}
Oct 30 01:25:53.092483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.089251    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" event={"ID":"39713818-355a-4c98-b1b0-9c7da92a6803","Type":"ContainerStarted","Data":"cd50deb74f9288828318e23e7603af4628d1a05ac180c258dc30972daa06ac00"}
Oct 30 01:25:53.108608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.096834    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7" event={"ID":"3370d60f-e20d-4420-b2cb-32d15d5e3e59","Type":"ContainerDied","Data":"35ba8de2c93a2bf446c06a1ea791ab1bdf753555c1201ce3eb78396e7f569ecc"}
Oct 30 01:25:53.108608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.096903    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ba8de2c93a2bf446c06a1ea791ab1bdf753555c1201ce3eb78396e7f569ecc"
Oct 30 01:25:53.108608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.096995    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"
Oct 30 01:25:53.132049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.131987    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" podStartSLOduration=68.13196619 podStartE2EDuration="1m8.13196619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:53.1277985 +0000 UTC m=+2594.675488988" watchObservedRunningTime="2026-10-30 01:25:53.13196619 +0000 UTC m=+2594.679656689"
Oct 30 01:25:53.970478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.954654    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:53.970478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.954830    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 01:25:53.970478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:53.954880    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 01:25:54.170358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.170321    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"]
Oct 30 01:25:54.171362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.171344    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3370d60f-e20d-4420-b2cb-32d15d5e3e59" containerName="revoke-keys"
Oct 30 01:25:54.171362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.171364    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3370d60f-e20d-4420-b2cb-32d15d5e3e59" containerName="revoke-keys"
Oct 30 01:25:54.171505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.171394    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy"
Oct 30 01:25:54.171505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.171399    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy"
Oct 30 01:25:54.171595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.171584    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3370d60f-e20d-4420-b2cb-32d15d5e3e59" containerName="revoke-keys"
Oct 30 01:25:54.171633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.171597    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c888e628-a915-4c08-b192-10976b6148cc" containerName="istio-proxy"
Oct 30 01:25:54.209482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.209331    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.219399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.219318    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"]
Oct 30 01:25:54.224381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.223899    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:25:54.235296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.235258    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:54.235496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.235295    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:25:54.235496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.235341    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:25:54.261630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.261594    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.261837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.261657    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n26j\" (UniqueName: \"kubernetes.io/projected/018cedd4-61d0-463a-a89b-2362bf769ec0-kube-api-access-6n26j\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.261960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.261926    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.364171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.364134    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.364401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.364210    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n26j\" (UniqueName: \"kubernetes.io/projected/018cedd4-61d0-463a-a89b-2362bf769ec0-kube-api-access-6n26j\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.364491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.364468    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.365150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.365120    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.366732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.366709    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.414836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.414797    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n26j\" (UniqueName: \"kubernetes.io/projected/018cedd4-61d0-463a-a89b-2362bf769ec0-kube-api-access-6n26j\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.533837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.533730    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:54.918121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.917695    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 01:25:54.918121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.917755    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 01:25:54.929184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:54.929137    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"]
Oct 30 01:25:55.143028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:55.142936    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls" event={"ID":"018cedd4-61d0-463a-a89b-2362bf769ec0","Type":"ContainerStarted","Data":"e3d8794ee1a0290c4edfa4199545f3e745944704ff74e705f132ff123eeefc06"}
Oct 30 01:25:55.803482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:55.797639    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:25:55.803482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:55.800607    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:25:55.923249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:55.923213    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 01:25:55.924918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:55.924881    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 01:25:56.229748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:56.229708    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls" event={"ID":"018cedd4-61d0-463a-a89b-2362bf769ec0","Type":"ContainerStarted","Data":"7f52eb0ac3f4a5b27799f5436c7f6c3b7c3796055aa7249b1e2788355e9e726f"}
Oct 30 01:25:56.268818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:56.268752    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls" podStartSLOduration=2.268733908 podStartE2EDuration="2.268733908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:56.265885101 +0000 UTC m=+2597.813575596" watchObservedRunningTime="2026-10-30 01:25:56.268733908 +0000 UTC m=+2597.816424406"
Oct 30 01:25:56.918343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:56.918302    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 01:25:56.918888 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:56.918375    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 01:25:57.242078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:57.242045    5104 generic.go:356] "Generic (PLEG): container finished" podID="018cedd4-61d0-463a-a89b-2362bf769ec0" containerID="7f52eb0ac3f4a5b27799f5436c7f6c3b7c3796055aa7249b1e2788355e9e726f" exitCode=0
Oct 30 01:25:57.242241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:57.242130    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls" event={"ID":"018cedd4-61d0-463a-a89b-2362bf769ec0","Type":"ContainerDied","Data":"7f52eb0ac3f4a5b27799f5436c7f6c3b7c3796055aa7249b1e2788355e9e726f"}
Oct 30 01:25:57.922109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:57.918966    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body=
Oct 30 01:25:57.922109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:57.919030    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused"
Oct 30 01:25:58.030853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.030808    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"]
Oct 30 01:25:58.041944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.041908    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cf0615-rsg6c"]
Oct 30 01:25:58.566404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.566380    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:58.599418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.599378    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9750ee0b-7611-40e2-81af-355a67d3634b" path="/var/lib/kubelet/pods/9750ee0b-7611-40e2-81af-355a67d3634b/volumes"
Oct 30 01:25:58.727791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.727757    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018cedd4-61d0-463a-a89b-2362bf769ec0-kube-api-access-6n26j\" (UniqueName: \"kubernetes.io/projected/018cedd4-61d0-463a-a89b-2362bf769ec0-kube-api-access-6n26j\") pod \"018cedd4-61d0-463a-a89b-2362bf769ec0\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") "
Oct 30 01:25:58.727948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.727835    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-xks-ca\") pod \"018cedd4-61d0-463a-a89b-2362bf769ec0\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") "
Oct 30 01:25:58.727948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.727918    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-openshift-ca\") pod \"018cedd4-61d0-463a-a89b-2362bf769ec0\" (UID: \"018cedd4-61d0-463a-a89b-2362bf769ec0\") "
Oct 30 01:25:58.728350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.728324    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-openshift-ca" pod "018cedd4-61d0-463a-a89b-2362bf769ec0" (UID: "018cedd4-61d0-463a-a89b-2362bf769ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:58.728959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.728933    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:58.731499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.730668    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-xks-ca" pod "018cedd4-61d0-463a-a89b-2362bf769ec0" (UID: "018cedd4-61d0-463a-a89b-2362bf769ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:58.731499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.730949    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018cedd4-61d0-463a-a89b-2362bf769ec0-kube-api-access-6n26j" pod "018cedd4-61d0-463a-a89b-2362bf769ec0" (UID: "018cedd4-61d0-463a-a89b-2362bf769ec0"). InnerVolumeSpecName "kube-api-access-6n26j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:58.831139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.831044    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n26j\" (UniqueName: \"kubernetes.io/projected/018cedd4-61d0-463a-a89b-2362bf769ec0-kube-api-access-6n26j\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:58.831139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.831079    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018cedd4-61d0-463a-a89b-2362bf769ec0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:25:58.925103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.925065    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:58.925576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.925524    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:58.926215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:58.926191    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:25:59.264283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.264252    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"
Oct 30 01:25:59.264519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.264278    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls" event={"ID":"018cedd4-61d0-463a-a89b-2362bf769ec0","Type":"ContainerDied","Data":"e3d8794ee1a0290c4edfa4199545f3e745944704ff74e705f132ff123eeefc06"}
Oct 30 01:25:59.264519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.264329    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d8794ee1a0290c4edfa4199545f3e745944704ff74e705f132ff123eeefc06"
Oct 30 01:25:59.319024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.318981    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"]
Oct 30 01:25:59.326856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.326817    5104 cpu_manager.go:414] "containerMap: removing container" podUID="018cedd4-61d0-463a-a89b-2362bf769ec0" containerName="revoke-keys"
Oct 30 01:25:59.326856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.326849    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018cedd4-61d0-463a-a89b-2362bf769ec0" containerName="revoke-keys"
Oct 30 01:25:59.327123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.327068    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="018cedd4-61d0-463a-a89b-2362bf769ec0" containerName="revoke-keys"
Oct 30 01:25:59.333265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.333231    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.336170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.336140    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"]
Oct 30 01:25:59.349502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.347328    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:25:59.349502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.347331    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:59.349502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.347532    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:25:59.349502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.347564    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:25:59.455214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.455161    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.455399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.455312    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.455586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.455564    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kpp\" (UniqueName: \"kubernetes.io/projected/57f74809-f096-4414-8e9d-c6fab705fe3d-kube-api-access-v2kpp\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.557706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.557615    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kpp\" (UniqueName: \"kubernetes.io/projected/57f74809-f096-4414-8e9d-c6fab705fe3d-kube-api-access-v2kpp\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.557881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.557822    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.557933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.557881    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.558665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.558640    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.560297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.560273    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.576286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.576255    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kpp\" (UniqueName: \"kubernetes.io/projected/57f74809-f096-4414-8e9d-c6fab705fe3d-kube-api-access-v2kpp\") pod \"maas-api-revoke-sub-349b4982-trjdx\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.652022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.651987    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:25:59.969570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:25:59.969530    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"]
Oct 30 01:25:59.970028 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:25:59.969944    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f74809_f096_4414_8e9d_c6fab705fe3d.slice/crio-45abbc5446e4cb64945580d3ee1569f387c4db252839c8ea3f333f688d2b57a5 WatchSource:0}: Error finding container 45abbc5446e4cb64945580d3ee1569f387c4db252839c8ea3f333f688d2b57a5: Status 404 returned error can't find the container with id 45abbc5446e4cb64945580d3ee1569f387c4db252839c8ea3f333f688d2b57a5
Oct 30 01:26:00.277662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:00.277576    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx" event={"ID":"57f74809-f096-4414-8e9d-c6fab705fe3d","Type":"ContainerStarted","Data":"45abbc5446e4cb64945580d3ee1569f387c4db252839c8ea3f333f688d2b57a5"}
Oct 30 01:26:01.290100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:01.290061    5104 generic.go:356] "Generic (PLEG): container finished" podID="57f74809-f096-4414-8e9d-c6fab705fe3d" containerID="838695f576db6cef86406bdd072fc9ecc02ba542c59f82574c5413169e577c13" exitCode=0
Oct 30 01:26:01.290621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:01.290148    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx" event={"ID":"57f74809-f096-4414-8e9d-c6fab705fe3d","Type":"ContainerDied","Data":"838695f576db6cef86406bdd072fc9ecc02ba542c59f82574c5413169e577c13"}
Oct 30 01:26:02.499137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.499108    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:26:02.515306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.515274    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-xks-ca\") pod \"57f74809-f096-4414-8e9d-c6fab705fe3d\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") "
Oct 30 01:26:02.515505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.515354    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-openshift-ca\") pod \"57f74809-f096-4414-8e9d-c6fab705fe3d\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") "
Oct 30 01:26:02.515608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.515526    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f74809-f096-4414-8e9d-c6fab705fe3d-kube-api-access-v2kpp\" (UniqueName: \"kubernetes.io/projected/57f74809-f096-4414-8e9d-c6fab705fe3d-kube-api-access-v2kpp\") pod \"57f74809-f096-4414-8e9d-c6fab705fe3d\" (UID: \"57f74809-f096-4414-8e9d-c6fab705fe3d\") "
Oct 30 01:26:02.515799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.515771    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-openshift-ca" pod "57f74809-f096-4414-8e9d-c6fab705fe3d" (UID: "57f74809-f096-4414-8e9d-c6fab705fe3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:02.516617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.516582    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:02.517554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.517525    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-xks-ca" pod "57f74809-f096-4414-8e9d-c6fab705fe3d" (UID: "57f74809-f096-4414-8e9d-c6fab705fe3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:02.518110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.518078    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f74809-f096-4414-8e9d-c6fab705fe3d-kube-api-access-v2kpp" pod "57f74809-f096-4414-8e9d-c6fab705fe3d" (UID: "57f74809-f096-4414-8e9d-c6fab705fe3d"). InnerVolumeSpecName "kube-api-access-v2kpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:02.618708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.618668    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2kpp\" (UniqueName: \"kubernetes.io/projected/57f74809-f096-4414-8e9d-c6fab705fe3d-kube-api-access-v2kpp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:02.618708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:02.618700    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f74809-f096-4414-8e9d-c6fab705fe3d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.310548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:03.310512    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"
Oct 30 01:26:03.310548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:03.310529    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx" event={"ID":"57f74809-f096-4414-8e9d-c6fab705fe3d","Type":"ContainerDied","Data":"45abbc5446e4cb64945580d3ee1569f387c4db252839c8ea3f333f688d2b57a5"}
Oct 30 01:26:03.310803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:03.310592    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45abbc5446e4cb64945580d3ee1569f387c4db252839c8ea3f333f688d2b57a5"
Oct 30 01:26:09.107200 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:09.107162    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:26:23.903426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.903384    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"]
Oct 30 01:26:23.904954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.904929    5104 cpu_manager.go:414] "containerMap: removing container" podUID="57f74809-f096-4414-8e9d-c6fab705fe3d" containerName="revoke-keys"
Oct 30 01:26:23.905041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.904956    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f74809-f096-4414-8e9d-c6fab705fe3d" containerName="revoke-keys"
Oct 30 01:26:23.905324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.905309    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f74809-f096-4414-8e9d-c6fab705fe3d" containerName="revoke-keys"
Oct 30 01:26:23.912019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.911989    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:23.923238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.923199    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"]
Oct 30 01:26:23.927588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.927561    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:23.927758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.927668    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:26:23.927758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.927720    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:26:23.938549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.938518    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:26:24.000486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:23.999924    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfkb\" (UniqueName: \"kubernetes.io/projected/a5e48b48-f963-4b81-9681-c60c0e838b74-kube-api-access-2bfkb\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.000486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.000229    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.000486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.000297    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.104474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.102302    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfkb\" (UniqueName: \"kubernetes.io/projected/a5e48b48-f963-4b81-9681-c60c0e838b74-kube-api-access-2bfkb\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.104474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.102564    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.104474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.102623    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.104474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.103568    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.110653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.105900    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.131683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.131634    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfkb\" (UniqueName: \"kubernetes.io/projected/a5e48b48-f963-4b81-9681-c60c0e838b74-kube-api-access-2bfkb\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:24.234737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:24.234698    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:25.331489 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:26:25.331404    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e48b48_f963_4b81_9681_c60c0e838b74.slice/crio-a0d9c95733679675032718389b431d29b8f0129007bfb6c0ff37cd3f70a838e7 WatchSource:0}: Error finding container a0d9c95733679675032718389b431d29b8f0129007bfb6c0ff37cd3f70a838e7: Status 404 returned error can't find the container with id a0d9c95733679675032718389b431d29b8f0129007bfb6c0ff37cd3f70a838e7
Oct 30 01:26:25.332395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:25.332372    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"]
Oct 30 01:26:26.071202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:26.071160    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" event={"ID":"a5e48b48-f963-4b81-9681-c60c0e838b74","Type":"ContainerStarted","Data":"8d71f9048bea20e8bdf13d6683960391c776c3456278f47ac9f17741aba5482d"}
Oct 30 01:26:26.071389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:26.071224    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" event={"ID":"a5e48b48-f963-4b81-9681-c60c0e838b74","Type":"ContainerStarted","Data":"a0d9c95733679675032718389b431d29b8f0129007bfb6c0ff37cd3f70a838e7"}
Oct 30 01:26:26.108148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:26.108088    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" podStartSLOduration=3.108072766 podStartE2EDuration="3.108072766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:26.106017402 +0000 UTC m=+2627.653707897" watchObservedRunningTime="2026-10-30 01:26:26.108072766 +0000 UTC m=+2627.655763272"
Oct 30 01:26:27.087506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:27.086612    5104 generic.go:356] "Generic (PLEG): container finished" podID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerID="8d71f9048bea20e8bdf13d6683960391c776c3456278f47ac9f17741aba5482d" exitCode=7
Oct 30 01:26:27.087506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:27.086732    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" event={"ID":"a5e48b48-f963-4b81-9681-c60c0e838b74","Type":"ContainerDied","Data":"8d71f9048bea20e8bdf13d6683960391c776c3456278f47ac9f17741aba5482d"}
Oct 30 01:26:27.087506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:27.087084    5104 scope.go:122] "RemoveContainer" containerID="8d71f9048bea20e8bdf13d6683960391c776c3456278f47ac9f17741aba5482d"
Oct 30 01:26:28.101691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:28.101657    5104 generic.go:356] "Generic (PLEG): container finished" podID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerID="70393d7afb45fb43388aa18e4be12278b2b62162e8180936f63308ce20be2d84" exitCode=7
Oct 30 01:26:28.102132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:28.101783    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" event={"ID":"a5e48b48-f963-4b81-9681-c60c0e838b74","Type":"ContainerDied","Data":"70393d7afb45fb43388aa18e4be12278b2b62162e8180936f63308ce20be2d84"}
Oct 30 01:26:28.102132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:28.101858    5104 scope.go:122] "RemoveContainer" containerID="8d71f9048bea20e8bdf13d6683960391c776c3456278f47ac9f17741aba5482d"
Oct 30 01:26:28.102397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:28.102377    5104 scope.go:122] "RemoveContainer" containerID="70393d7afb45fb43388aa18e4be12278b2b62162e8180936f63308ce20be2d84"
Oct 30 01:26:28.102914 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:28.102891    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c_odh-ai-gateway-infra(a5e48b48-f963-4b81-9681-c60c0e838b74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74"
Oct 30 01:26:28.654901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:28.654864    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:26:29.114180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.114138    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"]
Oct 30 01:26:29.119371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.119343    5104 scope.go:122] "RemoveContainer" containerID="70393d7afb45fb43388aa18e4be12278b2b62162e8180936f63308ce20be2d84"
Oct 30 01:26:29.119933 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:29.119897    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c_odh-ai-gateway-infra(a5e48b48-f963-4b81-9681-c60c0e838b74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74"
Oct 30 01:26:29.121813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.121787    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.135729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.135698    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"]
Oct 30 01:26:29.299154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.299101    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.299504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.299446    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.299504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.299491    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkr6b\" (UniqueName: \"kubernetes.io/projected/a282019e-ae1a-4d86-bbb9-12bfc98e6152-kube-api-access-jkr6b\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.401505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.401388    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.401505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.401443    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkr6b\" (UniqueName: \"kubernetes.io/projected/a282019e-ae1a-4d86-bbb9-12bfc98e6152-kube-api-access-jkr6b\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.401761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.401662    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.402327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.402306    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.403908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.403879    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.419767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.419733    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkr6b\" (UniqueName: \"kubernetes.io/projected/a282019e-ae1a-4d86-bbb9-12bfc98e6152-kube-api-access-jkr6b\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.442362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.442327    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:29.964501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:29.961205    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"]
Oct 30 01:26:30.149516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:30.148408    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerStarted","Data":"bdd2948a6aeb9f59ecb3379327c3df3c0d898c4d7b7aa81758a3091bc6658637"}
Oct 30 01:26:31.160150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:31.160111    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerStarted","Data":"c95ba0dcaa2361ed23750d7cebd1fea2391ef15ac92e5c0c750022949edb3c90"}
Oct 30 01:26:31.198250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:31.198198    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" podStartSLOduration=2.198182881 podStartE2EDuration="2.198182881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:31.19667783 +0000 UTC m=+2632.744368326" watchObservedRunningTime="2026-10-30 01:26:31.198182881 +0000 UTC m=+2632.745873377"
Oct 30 01:26:32.171975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:32.171934    5104 generic.go:356] "Generic (PLEG): container finished" podID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerID="c95ba0dcaa2361ed23750d7cebd1fea2391ef15ac92e5c0c750022949edb3c90" exitCode=7
Oct 30 01:26:32.172415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:32.172055    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerDied","Data":"c95ba0dcaa2361ed23750d7cebd1fea2391ef15ac92e5c0c750022949edb3c90"}
Oct 30 01:26:32.172582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:32.172511    5104 scope.go:122] "RemoveContainer" containerID="c95ba0dcaa2361ed23750d7cebd1fea2391ef15ac92e5c0c750022949edb3c90"
Oct 30 01:26:33.186842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:33.186801    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerStarted","Data":"45140f025ccfd0d50fdbb4ee5449f22d340dd28276cbf2e4df39d82d643bde7b"}
Oct 30 01:26:34.203650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:34.203613    5104 generic.go:356] "Generic (PLEG): container finished" podID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerID="45140f025ccfd0d50fdbb4ee5449f22d340dd28276cbf2e4df39d82d643bde7b" exitCode=7
Oct 30 01:26:34.204174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:34.203692    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerDied","Data":"45140f025ccfd0d50fdbb4ee5449f22d340dd28276cbf2e4df39d82d643bde7b"}
Oct 30 01:26:34.204174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:34.203755    5104 scope.go:122] "RemoveContainer" containerID="c95ba0dcaa2361ed23750d7cebd1fea2391ef15ac92e5c0c750022949edb3c90"
Oct 30 01:26:34.204174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:34.204168    5104 scope.go:122] "RemoveContainer" containerID="45140f025ccfd0d50fdbb4ee5449f22d340dd28276cbf2e4df39d82d643bde7b"
Oct 30 01:26:34.204779 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:34.204749    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg_odh-ai-gateway-infra(a282019e-ae1a-4d86-bbb9-12bfc98e6152)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152"
Oct 30 01:26:35.217366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:35.217336    5104 scope.go:122] "RemoveContainer" containerID="45140f025ccfd0d50fdbb4ee5449f22d340dd28276cbf2e4df39d82d643bde7b"
Oct 30 01:26:35.217851 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:35.217799    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg_odh-ai-gateway-infra(a282019e-ae1a-4d86-bbb9-12bfc98e6152)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152"
Oct 30 01:26:38.794510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:38.794435    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"]
Oct 30 01:26:38.795019 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:38.794690    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4" podUID="58db24d1-9c08-43f2-86f1-af1d5c66b110" containerName="payload-pre-processing" containerID="cri-o://e89f46f12d281aa3285fd65b298bc4d12c010e58bf836fc37bd71b428f4d78f4" gracePeriod=30
Oct 30 01:26:39.696327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.696288    5104 generic.go:356] "Generic (PLEG): container finished" podID="58db24d1-9c08-43f2-86f1-af1d5c66b110" containerID="e89f46f12d281aa3285fd65b298bc4d12c010e58bf836fc37bd71b428f4d78f4" exitCode=0
Oct 30 01:26:39.727548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.704347    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4" event={"ID":"58db24d1-9c08-43f2-86f1-af1d5c66b110","Type":"ContainerDied","Data":"e89f46f12d281aa3285fd65b298bc4d12c010e58bf836fc37bd71b428f4d78f4"}
Oct 30 01:26:39.743087 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:39.709870    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:26:39.843509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.810910    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"]
Oct 30 01:26:39.897920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.897784    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2a05c-vzv9k"]
Oct 30 01:26:39.937116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.937089    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:26:39.983945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.983844    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58db24d1-9c08-43f2-86f1-af1d5c66b110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58db24d1-9c08-43f2-86f1-af1d5c66b110-plugins-config-volume\") pod \"58db24d1-9c08-43f2-86f1-af1d5c66b110\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") "
Oct 30 01:26:39.984221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.983984    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58db24d1-9c08-43f2-86f1-af1d5c66b110-kube-api-access-v65s7\" (UniqueName: \"kubernetes.io/projected/58db24d1-9c08-43f2-86f1-af1d5c66b110-kube-api-access-v65s7\") pod \"58db24d1-9c08-43f2-86f1-af1d5c66b110\" (UID: \"58db24d1-9c08-43f2-86f1-af1d5c66b110\") "
Oct 30 01:26:39.984330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.984302    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58db24d1-9c08-43f2-86f1-af1d5c66b110-plugins-config-volume" pod "58db24d1-9c08-43f2-86f1-af1d5c66b110" (UID: "58db24d1-9c08-43f2-86f1-af1d5c66b110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:39.984964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.984935    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58db24d1-9c08-43f2-86f1-af1d5c66b110-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.986838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:39.986811    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58db24d1-9c08-43f2-86f1-af1d5c66b110-kube-api-access-v65s7" pod "58db24d1-9c08-43f2-86f1-af1d5c66b110" (UID: "58db24d1-9c08-43f2-86f1-af1d5c66b110"). InnerVolumeSpecName "kube-api-access-v65s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:40.087113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.087063    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v65s7\" (UniqueName: \"kubernetes.io/projected/58db24d1-9c08-43f2-86f1-af1d5c66b110-kube-api-access-v65s7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:40.570700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.570670    5104 scope.go:122] "RemoveContainer" containerID="70393d7afb45fb43388aa18e4be12278b2b62162e8180936f63308ce20be2d84"
Oct 30 01:26:40.585851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.585811    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf" path="/var/lib/kubelet/pods/afbdaf72-d41c-42d3-8cf9-8a6c898a8ecf/volumes"
Oct 30 01:26:40.781547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.780219    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4" event={"ID":"58db24d1-9c08-43f2-86f1-af1d5c66b110","Type":"ContainerDied","Data":"6d60835dc46bc596134f3d096eb0f1de3aa6c6611ba285770f39c87619decb08"}
Oct 30 01:26:40.781547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.780323    5104 scope.go:122] "RemoveContainer" containerID="e89f46f12d281aa3285fd65b298bc4d12c010e58bf836fc37bd71b428f4d78f4"
Oct 30 01:26:40.781547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.780533    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"
Oct 30 01:26:40.865479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.858603    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"]
Oct 30 01:26:40.869287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.869232    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4"]
Oct 30 01:26:40.901062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:40.901015    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2a05c-9d7464475-9p5s4" podUID="58db24d1-9c08-43f2-86f1-af1d5c66b110" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.1.94:9004: i/o timeout"
Oct 30 01:26:41.798161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:41.798125    5104 generic.go:356] "Generic (PLEG): container finished" podID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerID="27c51e61696c2d77fbc7e4c1496f519eaad71248b67c97c716bd0e51de3331cf" exitCode=7
Oct 30 01:26:41.798376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:41.798208    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" event={"ID":"a5e48b48-f963-4b81-9681-c60c0e838b74","Type":"ContainerDied","Data":"27c51e61696c2d77fbc7e4c1496f519eaad71248b67c97c716bd0e51de3331cf"}
Oct 30 01:26:41.798376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:41.798284    5104 scope.go:122] "RemoveContainer" containerID="70393d7afb45fb43388aa18e4be12278b2b62162e8180936f63308ce20be2d84"
Oct 30 01:26:41.799041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:41.798876    5104 scope.go:122] "RemoveContainer" containerID="27c51e61696c2d77fbc7e4c1496f519eaad71248b67c97c716bd0e51de3331cf"
Oct 30 01:26:41.799382 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:41.799357    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c_odh-ai-gateway-infra(a5e48b48-f963-4b81-9681-c60c0e838b74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74"
Oct 30 01:26:42.595044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:42.594999    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58db24d1-9c08-43f2-86f1-af1d5c66b110" path="/var/lib/kubelet/pods/58db24d1-9c08-43f2-86f1-af1d5c66b110/volumes"
Oct 30 01:26:42.850328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:42.850248    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"]
Oct 30 01:26:43.090832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.090799    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:43.185417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.185379    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-openshift-ca\") pod \"a5e48b48-f963-4b81-9681-c60c0e838b74\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") "
Oct 30 01:26:43.185680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.185576    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e48b48-f963-4b81-9681-c60c0e838b74-kube-api-access-2bfkb\" (UniqueName: \"kubernetes.io/projected/a5e48b48-f963-4b81-9681-c60c0e838b74-kube-api-access-2bfkb\") pod \"a5e48b48-f963-4b81-9681-c60c0e838b74\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") "
Oct 30 01:26:43.185771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.185752    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-xks-ca\") pod \"a5e48b48-f963-4b81-9681-c60c0e838b74\" (UID: \"a5e48b48-f963-4b81-9681-c60c0e838b74\") "
Oct 30 01:26:43.185950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.185916    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-openshift-ca" pod "a5e48b48-f963-4b81-9681-c60c0e838b74" (UID: "a5e48b48-f963-4b81-9681-c60c0e838b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:43.186830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.186803    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:43.189778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.188345    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-xks-ca" pod "a5e48b48-f963-4b81-9681-c60c0e838b74" (UID: "a5e48b48-f963-4b81-9681-c60c0e838b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:43.189778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.188380    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e48b48-f963-4b81-9681-c60c0e838b74-kube-api-access-2bfkb" pod "a5e48b48-f963-4b81-9681-c60c0e838b74" (UID: "a5e48b48-f963-4b81-9681-c60c0e838b74"). InnerVolumeSpecName "kube-api-access-2bfkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:43.314087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.293304    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bfkb\" (UniqueName: \"kubernetes.io/projected/a5e48b48-f963-4b81-9681-c60c0e838b74-kube-api-access-2bfkb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:43.314087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.293352    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e48b48-f963-4b81-9681-c60c0e838b74-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:43.823986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.823942    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"
Oct 30 01:26:43.823986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.823943    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c" event={"ID":"a5e48b48-f963-4b81-9681-c60c0e838b74","Type":"ContainerDied","Data":"a0d9c95733679675032718389b431d29b8f0129007bfb6c0ff37cd3f70a838e7"}
Oct 30 01:26:43.824590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.824025    5104 scope.go:122] "RemoveContainer" containerID="27c51e61696c2d77fbc7e4c1496f519eaad71248b67c97c716bd0e51de3331cf"
Oct 30 01:26:43.906478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.897404    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"]
Oct 30 01:26:43.911550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:43.911511    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-pzv5c"]
Oct 30 01:26:44.584155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:44.584113    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" path="/var/lib/kubelet/pods/a5e48b48-f963-4b81-9681-c60c0e838b74/volumes"
Oct 30 01:26:45.075836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.075789    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"]
Oct 30 01:26:45.077299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077270    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:45.077299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077297    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:45.077547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077316    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:45.077547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077324    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:45.077547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077380    5104 cpu_manager.go:414] "containerMap: removing container" podUID="58db24d1-9c08-43f2-86f1-af1d5c66b110" containerName="payload-pre-processing"
Oct 30 01:26:45.077547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077387    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58db24d1-9c08-43f2-86f1-af1d5c66b110" containerName="payload-pre-processing"
Oct 30 01:26:45.077792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077687    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:45.077792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077705    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:45.077792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.077725    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="58db24d1-9c08-43f2-86f1-af1d5c66b110" containerName="payload-pre-processing"
Oct 30 01:26:45.083781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.083758    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:45.100637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.100592    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"]
Oct 30 01:26:45.103118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.102219    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-29bb56\""
Oct 30 01:26:45.103118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.102355    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-29bb56-dockercfg-xvvrz\""
Oct 30 01:26:45.270543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.267857    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpg94\" (UniqueName: \"kubernetes.io/projected/eb0b22ab-d662-4495-ac26-f5ad3f84723c-kube-api-access-kpg94\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:45.270543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.267929    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0b22ab-d662-4495-ac26-f5ad3f84723c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:45.378531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.373067    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpg94\" (UniqueName: \"kubernetes.io/projected/eb0b22ab-d662-4495-ac26-f5ad3f84723c-kube-api-access-kpg94\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:45.378531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.373137    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0b22ab-d662-4495-ac26-f5ad3f84723c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:45.378531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.374341    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0b22ab-d662-4495-ac26-f5ad3f84723c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:45.612518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.612417    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpg94\" (UniqueName: \"kubernetes.io/projected/eb0b22ab-d662-4495-ac26-f5ad3f84723c-kube-api-access-kpg94\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:45.711747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:45.711705    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:46.063507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:46.057655    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"]
Oct 30 01:26:46.063507 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:26:46.060598    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0b22ab_d662_4495_ac26_f5ad3f84723c.slice/crio-e8bdcdf06f141a1881ca2892711bf6c4db5e60c22cb56101f671fd4453182a6e WatchSource:0}: Error finding container e8bdcdf06f141a1881ca2892711bf6c4db5e60c22cb56101f671fd4453182a6e: Status 404 returned error can't find the container with id e8bdcdf06f141a1881ca2892711bf6c4db5e60c22cb56101f671fd4453182a6e
Oct 30 01:26:46.583509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:46.580107    5104 scope.go:122] "RemoveContainer" containerID="45140f025ccfd0d50fdbb4ee5449f22d340dd28276cbf2e4df39d82d643bde7b"
Oct 30 01:26:47.249268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.249239    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"]
Oct 30 01:26:47.249405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.249309    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"]
Oct 30 01:26:47.249568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.249505    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy" containerID="cri-o://6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d" gracePeriod=30
Oct 30 01:26:47.249814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.249747    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy" containerID="cri-o://cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4" gracePeriod=30
Oct 30 01:26:47.252659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.252625    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s" event={"ID":"eb0b22ab-d662-4495-ac26-f5ad3f84723c","Type":"ContainerStarted","Data":"112a1b69b44f6e6f3895df7fac66085399411bf5311452881b378817f87cbbeb"}
Oct 30 01:26:47.252778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.252690    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s" event={"ID":"eb0b22ab-d662-4495-ac26-f5ad3f84723c","Type":"ContainerStarted","Data":"e8bdcdf06f141a1881ca2892711bf6c4db5e60c22cb56101f671fd4453182a6e"}
Oct 30 01:26:47.253756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.253730    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:26:47.255005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.254979    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"]
Oct 30 01:26:47.255279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.255239    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy" containerID="cri-o://57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e" gracePeriod=30
Oct 30 01:26:47.262262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.262234    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"]
Oct 30 01:26:47.263319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.263297    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:47.263319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.263319    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:47.263708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.263690    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e48b48-f963-4b81-9681-c60c0e838b74" containerName="revoke-keys"
Oct 30 01:26:47.270053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.270032    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"]
Oct 30 01:26:47.270174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.270137    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.427479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.424571    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdj5q\" (UniqueName: \"kubernetes.io/projected/7ae5658b-b2dd-4684-a50f-52abed8430e3-kube-api-access-rdj5q\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.427479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.424799    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.427479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.425081    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.450596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.450538    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s" podStartSLOduration=2.450521572 podStartE2EDuration="2.450521572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:47.448735966 +0000 UTC m=+2648.996426481" watchObservedRunningTime="2026-10-30 01:26:47.450521572 +0000 UTC m=+2648.998212069"
Oct 30 01:26:47.526551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.526439    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.526738 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.526595    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdj5q\" (UniqueName: \"kubernetes.io/projected/7ae5658b-b2dd-4684-a50f-52abed8430e3-kube-api-access-rdj5q\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.526827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.526770    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.527273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.527243    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.529389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.529366    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.544723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.544690    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdj5q\" (UniqueName: \"kubernetes.io/projected/7ae5658b-b2dd-4684-a50f-52abed8430e3-kube-api-access-rdj5q\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:47.604487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:47.604157    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:48.035992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:48.035960    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"]
Oct 30 01:26:48.036962 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:26:48.036935    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae5658b_b2dd_4684_a50f_52abed8430e3.slice/crio-1e33382ce76ca954e223c4d70842ff163e706e1bbf1f2ab24846dbef76155b9f WatchSource:0}: Error finding container 1e33382ce76ca954e223c4d70842ff163e706e1bbf1f2ab24846dbef76155b9f: Status 404 returned error can't find the container with id 1e33382ce76ca954e223c4d70842ff163e706e1bbf1f2ab24846dbef76155b9f
Oct 30 01:26:48.265727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:48.265693    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerStarted","Data":"c2a6f65598ae3ae614b16acca2dc19f99f424626c8b546dfe92e4b219ccf4d0d"}
Oct 30 01:26:48.267392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:48.267361    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs" event={"ID":"7ae5658b-b2dd-4684-a50f-52abed8430e3","Type":"ContainerStarted","Data":"1e33382ce76ca954e223c4d70842ff163e706e1bbf1f2ab24846dbef76155b9f"}
Oct 30 01:26:49.281646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:49.281610    5104 generic.go:356] "Generic (PLEG): container finished" podID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerID="c2a6f65598ae3ae614b16acca2dc19f99f424626c8b546dfe92e4b219ccf4d0d" exitCode=7
Oct 30 01:26:49.282154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:49.281683    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerDied","Data":"c2a6f65598ae3ae614b16acca2dc19f99f424626c8b546dfe92e4b219ccf4d0d"}
Oct 30 01:26:49.282154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:49.281748    5104 scope.go:122] "RemoveContainer" containerID="45140f025ccfd0d50fdbb4ee5449f22d340dd28276cbf2e4df39d82d643bde7b"
Oct 30 01:26:49.282277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:49.282236    5104 scope.go:122] "RemoveContainer" containerID="c2a6f65598ae3ae614b16acca2dc19f99f424626c8b546dfe92e4b219ccf4d0d"
Oct 30 01:26:49.282751 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:49.282723    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg_odh-ai-gateway-infra(a282019e-ae1a-4d86-bbb9-12bfc98e6152)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152"
Oct 30 01:26:49.284705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:49.284679    5104 generic.go:356] "Generic (PLEG): container finished" podID="7ae5658b-b2dd-4684-a50f-52abed8430e3" containerID="9d1419047bfc583340709b24947da7cd535308a3ba9b30acddb9efca29f0f8ea" exitCode=0
Oct 30 01:26:49.285994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:49.285968    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs" event={"ID":"7ae5658b-b2dd-4684-a50f-52abed8430e3","Type":"ContainerDied","Data":"9d1419047bfc583340709b24947da7cd535308a3ba9b30acddb9efca29f0f8ea"}
Oct 30 01:26:49.316079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:49.316041    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"]
Oct 30 01:26:50.510175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.510141    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:50.586578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.586543    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:50.692664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.692633    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-xks-ca\") pod \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") "
Oct 30 01:26:50.692885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.692728    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a282019e-ae1a-4d86-bbb9-12bfc98e6152-kube-api-access-jkr6b\" (UniqueName: \"kubernetes.io/projected/a282019e-ae1a-4d86-bbb9-12bfc98e6152-kube-api-access-jkr6b\") pod \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") "
Oct 30 01:26:50.692885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.692842    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-xks-ca\") pod \"7ae5658b-b2dd-4684-a50f-52abed8430e3\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") "
Oct 30 01:26:50.693027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.692998    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-openshift-ca\") pod \"7ae5658b-b2dd-4684-a50f-52abed8430e3\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") "
Oct 30 01:26:50.693159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.693140    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-openshift-ca\") pod \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\" (UID: \"a282019e-ae1a-4d86-bbb9-12bfc98e6152\") "
Oct 30 01:26:50.693467 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.693420    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-openshift-ca" pod "7ae5658b-b2dd-4684-a50f-52abed8430e3" (UID: "7ae5658b-b2dd-4684-a50f-52abed8430e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:50.693467 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.693440    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae5658b-b2dd-4684-a50f-52abed8430e3-kube-api-access-rdj5q\" (UniqueName: \"kubernetes.io/projected/7ae5658b-b2dd-4684-a50f-52abed8430e3-kube-api-access-rdj5q\") pod \"7ae5658b-b2dd-4684-a50f-52abed8430e3\" (UID: \"7ae5658b-b2dd-4684-a50f-52abed8430e3\") "
Oct 30 01:26:50.693645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.693566    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-openshift-ca" pod "a282019e-ae1a-4d86-bbb9-12bfc98e6152" (UID: "a282019e-ae1a-4d86-bbb9-12bfc98e6152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:50.695024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.694793    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:50.695024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.694819    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:50.696583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.696555    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-xks-ca" pod "7ae5658b-b2dd-4684-a50f-52abed8430e3" (UID: "7ae5658b-b2dd-4684-a50f-52abed8430e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:50.696730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.696705    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a282019e-ae1a-4d86-bbb9-12bfc98e6152-kube-api-access-jkr6b" pod "a282019e-ae1a-4d86-bbb9-12bfc98e6152" (UID: "a282019e-ae1a-4d86-bbb9-12bfc98e6152"). InnerVolumeSpecName "kube-api-access-jkr6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:50.696843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.696824    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae5658b-b2dd-4684-a50f-52abed8430e3-kube-api-access-rdj5q" pod "7ae5658b-b2dd-4684-a50f-52abed8430e3" (UID: "7ae5658b-b2dd-4684-a50f-52abed8430e3"). InnerVolumeSpecName "kube-api-access-rdj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:50.697334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.697315    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-xks-ca" pod "a282019e-ae1a-4d86-bbb9-12bfc98e6152" (UID: "a282019e-ae1a-4d86-bbb9-12bfc98e6152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:50.796973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.796937    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkr6b\" (UniqueName: \"kubernetes.io/projected/a282019e-ae1a-4d86-bbb9-12bfc98e6152-kube-api-access-jkr6b\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:50.796973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.796969    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae5658b-b2dd-4684-a50f-52abed8430e3-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:50.796973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.796979    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdj5q\" (UniqueName: \"kubernetes.io/projected/7ae5658b-b2dd-4684-a50f-52abed8430e3-kube-api-access-rdj5q\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:50.797345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:50.796988    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a282019e-ae1a-4d86-bbb9-12bfc98e6152-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:51.308021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.307989    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"
Oct 30 01:26:51.308021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.308010    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs" event={"ID":"7ae5658b-b2dd-4684-a50f-52abed8430e3","Type":"ContainerDied","Data":"1e33382ce76ca954e223c4d70842ff163e706e1bbf1f2ab24846dbef76155b9f"}
Oct 30 01:26:51.308274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.308064    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e33382ce76ca954e223c4d70842ff163e706e1bbf1f2ab24846dbef76155b9f"
Oct 30 01:26:51.310039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.310014    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"
Oct 30 01:26:51.310196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.310046    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg" event={"ID":"a282019e-ae1a-4d86-bbb9-12bfc98e6152","Type":"ContainerDied","Data":"bdd2948a6aeb9f59ecb3379327c3df3c0d898c4d7b7aa81758a3091bc6658637"}
Oct 30 01:26:51.310196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.310095    5104 scope.go:122] "RemoveContainer" containerID="c2a6f65598ae3ae614b16acca2dc19f99f424626c8b546dfe92e4b219ccf4d0d"
Oct 30 01:26:51.364341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.364295    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"]
Oct 30 01:26:51.369854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:51.369822    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-nbdwg"]
Oct 30 01:26:52.593771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.593699    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" path="/var/lib/kubelet/pods/a282019e-ae1a-4d86-bbb9-12bfc98e6152/volumes"
Oct 30 01:26:52.619849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.619541    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:26:52.636614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.636578    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-socket\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.636897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.636700    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-podinfo\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.636897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.636768    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istiod-ca-cert\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.636897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.636838    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-data\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.637192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.636922    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-socket" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.637192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.636988    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-envoy\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.637192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.637067    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-token\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-token\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.637192 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.637157    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-kube-api-access-pb2jg\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-kube-api-access-pb2jg\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.637469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.637214    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-credential-socket\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.637469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.637293    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-certs\") pod \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\" (UID: \"75cf53c2-ab95-4e5a-a404-4ed17a140a1c\") "
Oct 30 01:26:52.641187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.638186    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-certs" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.641187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.638627    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.641187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.638651    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.641187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.639702    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-credential-socket" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.641187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.639890    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istiod-ca-cert" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:52.641187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.639957    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-data" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.641805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.641364    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-podinfo" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:52.641805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.641570    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-envoy" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.642846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.642773    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-token" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:52.643474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.643430    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-kube-api-access-pb2jg" pod "75cf53c2-ab95-4e5a-a404-4ed17a140a1c" (UID: "75cf53c2-ab95-4e5a-a404-4ed17a140a1c"). InnerVolumeSpecName "kube-api-access-pb2jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:52.697498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.683864    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.829734    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-kube-api-access-4vfkk\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-kube-api-access-4vfkk\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.829848    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-credential-socket\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.829932    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-podinfo\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.830057    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-data\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.830094    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-socket\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.830268    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-envoy\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.830339    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-certs\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.830389    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-token\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-token\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.830567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.830427    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ec7f91-11f2-43f8-93ce-04b319412bd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ec7f91-11f2-43f8-93ce-04b319412bd2-istiod-ca-cert\") pod \"99ec7f91-11f2-43f8-93ce-04b319412bd2\" (UID: \"99ec7f91-11f2-43f8-93ce-04b319412bd2\") "
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.846424    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.846478    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb2jg\" (UniqueName: \"kubernetes.io/projected/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-kube-api-access-pb2jg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.846496    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.846512    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.846534    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.846550    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.846566    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf53c2-ab95-4e5a-a404-4ed17a140a1c-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.847033    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ec7f91-11f2-43f8-93ce-04b319412bd2-istiod-ca-cert" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.849751    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-certs" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.850936    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-credential-socket" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.852980    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-data" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.853229    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-socket" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.857989    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-kube-api-access-4vfkk" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "kube-api-access-4vfkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.860787    5104 scope.go:122] "RemoveContainer" containerID="1ae419b5009d2a85044fe8e62ea227b4a7504b4aecf8920581dc1a1d0d7e4785"
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.861171    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-token" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:52.878135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.869699    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-envoy" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:52.890209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.890176    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"]
Oct 30 01:26:52.892916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.892888    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7ae5658b-b2dd-4684-a50f-52abed8430e3" containerName="revoke-keys"
Oct 30 01:26:52.893088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893071    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae5658b-b2dd-4684-a50f-52abed8430e3" containerName="revoke-keys"
Oct 30 01:26:52.893217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893204    5104 cpu_manager.go:414] "containerMap: removing container" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy"
Oct 30 01:26:52.893315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893303    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy"
Oct 30 01:26:52.893432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893419    5104 cpu_manager.go:414] "containerMap: removing container" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy"
Oct 30 01:26:52.893579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893558    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy"
Oct 30 01:26:52.893759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893744    5104 cpu_manager.go:414] "containerMap: removing container" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy"
Oct 30 01:26:52.893889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893875    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy"
Oct 30 01:26:52.893997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.893987    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.894076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894066    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.894202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894192    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.894271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894261    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.896513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894611    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-podinfo" pod "99ec7f91-11f2-43f8-93ce-04b319412bd2" (UID: "99ec7f91-11f2-43f8-93ce-04b319412bd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:52.896513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894922    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.896513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894957    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.896513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894978    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae5658b-b2dd-4684-a50f-52abed8430e3" containerName="revoke-keys"
Oct 30 01:26:52.896513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.894995    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerName="istio-proxy"
Oct 30 01:26:52.896513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.895005    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy"
Oct 30 01:26:52.896513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.895034    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerName="istio-proxy"
Oct 30 01:26:52.898722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.898696    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:26:52.902674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.902653    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:52.917799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.917767    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"]
Oct 30 01:26:52.941027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.940992    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"]
Oct 30 01:26:52.942589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.942561    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.942589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.942589    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.942770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.942631    5104 cpu_manager.go:414] "containerMap: removing container" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy"
Oct 30 01:26:52.942770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.942639    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy"
Oct 30 01:26:52.942956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.942934    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" containerName="istio-proxy"
Oct 30 01:26:52.943029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.942963    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a282019e-ae1a-4d86-bbb9-12bfc98e6152" containerName="revoke-keys"
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948565    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948596    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948612    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948626    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948641    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948653    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948665    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ec7f91-11f2-43f8-93ce-04b319412bd2-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948708    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfkk\" (UniqueName: \"kubernetes.io/projected/99ec7f91-11f2-43f8-93ce-04b319412bd2-kube-api-access-4vfkk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.948723    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ec7f91-11f2-43f8-93ce-04b319412bd2-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.953050    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:52.968124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.956736    5104 scope.go:122] "RemoveContainer" containerID="74c4e9de296b0ce5d7cf9cf75da7058ab3d2048ee8992f8699d3679d32e34e9f"
Oct 30 01:26:52.993875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.993848    5104 scope.go:122] "RemoveContainer" containerID="8549f3acd01abc886c4d71f6060508c87096eb2a6a79131206d55719fa499682"
Oct 30 01:26:52.996358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:52.996281    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"]
Oct 30 01:26:53.017421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.015826    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"]
Oct 30 01:26:53.022872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.022839    5104 scope.go:122] "RemoveContainer" containerID="6530257aadc76e6ddc1a608ffd84f9f7be6f7ecadcad2d47be8874074d473743"
Oct 30 01:26:53.025540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.025344    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.029079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.029058    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"]
Oct 30 01:26:53.043138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.041056    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nss4l\""
Oct 30 01:26:53.043138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.041822    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d3ed60-serving-cert\""
Oct 30 01:26:53.043138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.041895    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:26:53.043138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.041978    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:26:53.043138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.042084    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:26:53.050095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050051    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-credential-socket\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.050446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050278    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-kube-api-access-lphct\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-kube-api-access-lphct\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.050446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050337    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39713818-355a-4c98-b1b0-9c7da92a6803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39713818-355a-4c98-b1b0-9c7da92a6803-istio-podinfo\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.050446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050361    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-credential-socket" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:53.050694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050543    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-certs\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.050694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050571    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-socket\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.050694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050658    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-istio-token\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-istio-token\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.051338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050860    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-data\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.051338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050951    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-envoy\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.051338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.050994    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39713818-355a-4c98-b1b0-9c7da92a6803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39713818-355a-4c98-b1b0-9c7da92a6803-istiod-ca-cert\") pod \"39713818-355a-4c98-b1b0-9c7da92a6803\" (UID: \"39713818-355a-4c98-b1b0-9c7da92a6803\") "
Oct 30 01:26:53.051338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051113    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-certs" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:53.051338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051254    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-data" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:53.051338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051314    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-socket" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051492    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxwj\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-kube-api-access-bzxwj\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051560    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051682    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051729    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39713818-355a-4c98-b1b0-9c7da92a6803-istiod-ca-cert" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051756    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051859    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051892    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.051926    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052069    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052105    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052162    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.052251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052216    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.053040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052313    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.053040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052403    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.053040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052489    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.053040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052549    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.053040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052851    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.053040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052903    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.053040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.052960    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8p9k\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-kube-api-access-t8p9k\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.053434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.053195    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.053434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.053218    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.053434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.053233    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.053434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.053247    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39713818-355a-4c98-b1b0-9c7da92a6803-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.053434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.053261    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.053434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.053243    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-kube-api-access-lphct" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "kube-api-access-lphct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:53.053919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.053619    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39713818-355a-4c98-b1b0-9c7da92a6803-istio-podinfo" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:53.054265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.054232    5104 scope.go:122] "RemoveContainer" containerID="bf549b5eba9aa074c222c2d015687b60bb7f96c00f6567d732c158f44420cf34"
Oct 30 01:26:53.054989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.054541    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-istio-token" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:53.055336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.055259    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-envoy" pod "39713818-355a-4c98-b1b0-9c7da92a6803" (UID: "39713818-355a-4c98-b1b0-9c7da92a6803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:53.069211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.069190    5104 scope.go:122] "RemoveContainer" containerID="b05a34828ccb7d39ffcbee11e29ad212d4161cba93e10c7edc91b90118eb4654"
Oct 30 01:26:53.154641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.154601    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.154821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.154675    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.154821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.154741    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-metrics-certs\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.154821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.154794    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.154999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.154871    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.154999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.154928    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.154999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.154953    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.155174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155005    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.155174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155134    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.155284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155174    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.155284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155207    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.155284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155254    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8p9k\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-kube-api-access-t8p9k\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.155473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155306    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxwj\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-kube-api-access-bzxwj\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.155473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155349    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.155473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155377    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmsgn\" (UniqueName: \"kubernetes.io/projected/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-kube-api-access-dmsgn\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.155473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155426    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.155773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155444    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.155921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155848    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.155921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.155842    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.156499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.156405    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.158011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.157910    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.157995    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158042    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158056    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158142    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158174    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158206    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.158423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158275    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.158423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158298    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.158423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158325    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158539    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.158667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158583    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.158800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158683    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.158800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158727    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39713818-355a-4c98-b1b0-9c7da92a6803-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.158800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158746    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lphct\" (UniqueName: \"kubernetes.io/projected/39713818-355a-4c98-b1b0-9c7da92a6803-kube-api-access-lphct\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.158800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158758    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39713818-355a-4c98-b1b0-9c7da92a6803-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.158986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.158808    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.159095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.159073    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.159199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.159181    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.159879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.159856    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.175249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.175210    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8p9k\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-kube-api-access-t8p9k\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.175419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.175354    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxwj\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-kube-api-access-bzxwj\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.175419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.175364    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.176099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.176079    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.250232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.249927    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:53.285499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.261322    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmsgn\" (UniqueName: \"kubernetes.io/projected/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-kube-api-access-dmsgn\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.285499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.261718    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-metrics-certs\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.285499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.261913    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.285499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.262111    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.285499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.263298    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.288592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.288559    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:53.300039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.290853    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-metrics-certs\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.300039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.294113    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.316665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.312564    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmsgn\" (UniqueName: \"kubernetes.io/projected/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-kube-api-access-dmsgn\") pod \"maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.340124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.340006    5104 generic.go:356] "Generic (PLEG): container finished" podID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" containerID="cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4" exitCode=0
Oct 30 01:26:53.340292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.340140    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"
Oct 30 01:26:53.340644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.340159    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" event={"ID":"75cf53c2-ab95-4e5a-a404-4ed17a140a1c","Type":"ContainerDied","Data":"cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4"}
Oct 30 01:26:53.340644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.340594    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6" event={"ID":"75cf53c2-ab95-4e5a-a404-4ed17a140a1c","Type":"ContainerDied","Data":"d4d30e657e3aab6506c1701f701f27a5c4449c4c16d6a37c16badcf9ee535ae6"}
Oct 30 01:26:53.340644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.340636    5104 scope.go:122] "RemoveContainer" containerID="cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4"
Oct 30 01:26:53.344238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.344175    5104 generic.go:356] "Generic (PLEG): container finished" podID="99ec7f91-11f2-43f8-93ce-04b319412bd2" containerID="6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d" exitCode=0
Oct 30 01:26:53.344566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.344539    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" event={"ID":"99ec7f91-11f2-43f8-93ce-04b319412bd2","Type":"ContainerDied","Data":"6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d"}
Oct 30 01:26:53.344652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.344595    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj" event={"ID":"99ec7f91-11f2-43f8-93ce-04b319412bd2","Type":"ContainerDied","Data":"706df6737a1a82f8e863b946b10bf6d3dbb5fab9317be1a2aa54f4b0f4604f8c"}
Oct 30 01:26:53.344652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.344635    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"
Oct 30 01:26:53.348098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.348069    5104 generic.go:356] "Generic (PLEG): container finished" podID="39713818-355a-4c98-b1b0-9c7da92a6803" containerID="57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e" exitCode=0
Oct 30 01:26:53.348210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.348166    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"
Oct 30 01:26:53.348210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.348174    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" event={"ID":"39713818-355a-4c98-b1b0-9c7da92a6803","Type":"ContainerDied","Data":"57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e"}
Oct 30 01:26:53.348353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.348239    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq" event={"ID":"39713818-355a-4c98-b1b0-9c7da92a6803","Type":"ContainerDied","Data":"cd50deb74f9288828318e23e7603af4628d1a05ac180c258dc30972daa06ac00"}
Oct 30 01:26:53.356627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.356596    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:26:53.357277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.357250    5104 scope.go:122] "RemoveContainer" containerID="3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e"
Oct 30 01:26:53.414771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.414597    5104 scope.go:122] "RemoveContainer" containerID="cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4"
Oct 30 01:26:53.415195 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:53.415086    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4\": container with ID starting with cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4 not found: ID does not exist" containerID="cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4"
Oct 30 01:26:53.415195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.415128    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4"} err="failed to get container status \"cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4\": rpc error: code = NotFound desc = could not find container \"cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4\": container with ID starting with cb273e641bedb38faa4c91f39281979b1569ade3638097da0996b30975e5b3c4 not found: ID does not exist"
Oct 30 01:26:53.415195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.415160    5104 scope.go:122] "RemoveContainer" containerID="3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e"
Oct 30 01:26:53.415974 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:53.415909    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e\": container with ID starting with 3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e not found: ID does not exist" containerID="3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e"
Oct 30 01:26:53.416086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.415987    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e"} err="failed to get container status \"3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e\": rpc error: code = NotFound desc = could not find container \"3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e\": container with ID starting with 3bfb2e5fa08439c6f4f3eec2c3b49e53a5c35261453b70e63e18a15f2fa22b7e not found: ID does not exist"
Oct 30 01:26:53.416086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.416036    5104 scope.go:122] "RemoveContainer" containerID="6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d"
Oct 30 01:26:53.433814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.433764    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"]
Oct 30 01:26:53.441174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.441060    5104 scope.go:122] "RemoveContainer" containerID="6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d"
Oct 30 01:26:53.442227 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:53.442056    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d\": container with ID starting with 6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d not found: ID does not exist" containerID="6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d"
Oct 30 01:26:53.442227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.442098    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d"} err="failed to get container status \"6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d\": rpc error: code = NotFound desc = could not find container \"6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d\": container with ID starting with 6414e51cbcc471be3f506bfdd305ac8af03329239c1f6f87bcb17dea1889267d not found: ID does not exist"
Oct 30 01:26:53.442227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.442124    5104 scope.go:122] "RemoveContainer" containerID="57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e"
Oct 30 01:26:53.446637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.446611    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-hbdwq"]
Oct 30 01:26:53.467945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.467849    5104 scope.go:122] "RemoveContainer" containerID="57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e"
Oct 30 01:26:53.468325 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:26:53.468270    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e\": container with ID starting with 57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e not found: ID does not exist" containerID="57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e"
Oct 30 01:26:53.468508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.468315    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e"} err="failed to get container status \"57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e\": rpc error: code = NotFound desc = could not find container \"57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e\": container with ID starting with 57bc12e73fc86ce01660b88521f0bb0d214a0102724f8cb576f6e349fae87e6e not found: ID does not exist"
Oct 30 01:26:53.473416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.473393    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"]
Oct 30 01:26:53.481524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.481492    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-dhwd6"]
Oct 30 01:26:53.516290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.516252    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"]
Oct 30 01:26:53.527439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.527401    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-ph9vj"]
Oct 30 01:26:53.660882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.660838    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"]
Oct 30 01:26:53.664513 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:26:53.662673    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1bf6a4_4e23_46b6_bab8_74f806685b90.slice/crio-b571be44251caa17d8ed67386c855d5f2e94a4f58215d061764265345609d7ee WatchSource:0}: Error finding container b571be44251caa17d8ed67386c855d5f2e94a4f58215d061764265345609d7ee: Status 404 returned error can't find the container with id b571be44251caa17d8ed67386c855d5f2e94a4f58215d061764265345609d7ee
Oct 30 01:26:53.672341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.671119    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"]
Oct 30 01:26:53.673013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.672978    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:53.673129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.673088    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:53.673362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.673137    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:53.676044 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:26:53.676003    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52829d0c_ebfe_4b80_b611_c3b42b1c0a2c.slice/crio-db28d5df2e13851ce0fa698617781b00b213db6959ac0aaeedb14fec30b79b3c WatchSource:0}: Error finding container db28d5df2e13851ce0fa698617781b00b213db6959ac0aaeedb14fec30b79b3c: Status 404 returned error can't find the container with id db28d5df2e13851ce0fa698617781b00b213db6959ac0aaeedb14fec30b79b3c
Oct 30 01:26:53.686622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.686584    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:53.686766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.686660    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:53.686766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.686698    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:26:53.828766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:53.828702    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"]
Oct 30 01:26:54.391540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.391424    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" event={"ID":"4c1bf6a4-4e23-46b6-bab8-74f806685b90","Type":"ContainerStarted","Data":"c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667"}
Oct 30 01:26:54.391873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.391847    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" event={"ID":"4c1bf6a4-4e23-46b6-bab8-74f806685b90","Type":"ContainerStarted","Data":"b571be44251caa17d8ed67386c855d5f2e94a4f58215d061764265345609d7ee"}
Oct 30 01:26:54.396543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.396506    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" event={"ID":"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c","Type":"ContainerStarted","Data":"1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517"}
Oct 30 01:26:54.396788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.396767    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" event={"ID":"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c","Type":"ContainerStarted","Data":"db28d5df2e13851ce0fa698617781b00b213db6959ac0aaeedb14fec30b79b3c"}
Oct 30 01:26:54.434144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.434100    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442" event={"ID":"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0","Type":"ContainerStarted","Data":"9e36943f547ffc4cac600fc9f8f6d6e8f984f2b0204d19133f4b073c780de6d4"}
Oct 30 01:26:54.460995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.460930    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podStartSLOduration=123.460899639 podStartE2EDuration="2m3.460899639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:54.450672917 +0000 UTC m=+2655.998363412" watchObservedRunningTime="2026-10-30 01:26:54.460899639 +0000 UTC m=+2656.008590136"
Oct 30 01:26:54.501479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.500491    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podStartSLOduration=123.500468942 podStartE2EDuration="2m3.500468942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:54.495339133 +0000 UTC m=+2656.043029630" watchObservedRunningTime="2026-10-30 01:26:54.500468942 +0000 UTC m=+2656.048159441"
Oct 30 01:26:54.591886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.591845    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39713818-355a-4c98-b1b0-9c7da92a6803" path="/var/lib/kubelet/pods/39713818-355a-4c98-b1b0-9c7da92a6803/volumes"
Oct 30 01:26:54.593009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.592986    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cf53c2-ab95-4e5a-a404-4ed17a140a1c" path="/var/lib/kubelet/pods/75cf53c2-ab95-4e5a-a404-4ed17a140a1c/volumes"
Oct 30 01:26:54.593860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:54.593841    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ec7f91-11f2-43f8-93ce-04b319412bd2" path="/var/lib/kubelet/pods/99ec7f91-11f2-43f8-93ce-04b319412bd2/volumes"
Oct 30 01:26:55.251573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:55.251535    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:26:55.258371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:55.253823    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Oct 30 01:26:55.258371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:55.253885    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Oct 30 01:26:55.309512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:55.308576    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:26:55.310955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:55.310590    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 30 01:26:55.310955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:55.310654    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 30 01:26:56.251117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:56.251078    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Oct 30 01:26:56.251306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:56.251136    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Oct 30 01:26:56.331386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:56.331348    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 30 01:26:56.332496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:56.332058    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 30 01:26:57.253515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:57.253475    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Oct 30 01:26:57.253840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:57.253801    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Oct 30 01:26:57.299283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:57.299246    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 30 01:26:57.299591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:57.299557    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 30 01:26:58.251960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:58.251919    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Oct 30 01:26:58.252506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:58.252041    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Oct 30 01:26:58.333481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:58.324568    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 30 01:26:58.333481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:58.324648    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 30 01:26:59.251752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:59.251714    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Oct 30 01:26:59.252294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:59.252032    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Oct 30 01:26:59.297631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:59.296425    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 30 01:26:59.297631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:59.296518    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 30 01:26:59.336516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:26:59.334472    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:27:00.251767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:00.251446    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Oct 30 01:27:00.251767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:00.251530    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Oct 30 01:27:00.305484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:00.290966    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 30 01:27:00.305484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:00.291027    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 30 01:27:01.251446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:01.251406    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body=
Oct 30 01:27:01.251788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:01.251497    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused"
Oct 30 01:27:01.300417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:01.300283    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 30 01:27:01.300417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:01.300345    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 30 01:27:02.308845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.308808    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:27:02.310391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.310321    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:27:02.312038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.311973    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:27:02.409158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.409120    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:27:02.412917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.412886    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:27:02.414616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.414590    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:27:02.883890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.883857    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442" event={"ID":"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0","Type":"ContainerStarted","Data":"75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c"}
Oct 30 01:27:02.884112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.883949    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:27:02.922833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:02.922782    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442" podStartSLOduration=161.574986438 podStartE2EDuration="2m49.922767239s" totalImagesPullingTime="8.347780801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:02.920707595 +0000 UTC m=+2664.468398093" watchObservedRunningTime="2026-10-30 01:27:02.922767239 +0000 UTC m=+2664.470457734"
Oct 30 01:27:03.162786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.162690    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"]
Oct 30 01:27:03.174564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.169793    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.174564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.170108    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"]
Oct 30 01:27:03.190385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.190133    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:27:03.192883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.190880    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:03.265958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.265923    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.266187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.266163    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.266236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.266203    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8wn\" (UniqueName: \"kubernetes.io/projected/1efd82de-6184-4bb7-b070-5595a62c1284-kube-api-access-5v8wn\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.368441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.368405    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.368441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.368449    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8wn\" (UniqueName: \"kubernetes.io/projected/1efd82de-6184-4bb7-b070-5595a62c1284-kube-api-access-5v8wn\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.369024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.368715    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.369529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.369507    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.370898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.370879    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.387275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.387235    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8wn\" (UniqueName: \"kubernetes.io/projected/1efd82de-6184-4bb7-b070-5595a62c1284-kube-api-access-5v8wn\") pod \"maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.510586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.510546    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:03.888938 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:27:03.887666    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efd82de_6184_4bb7_b070_5595a62c1284.slice/crio-9cd0b5110ce17d9e4a03e094726497d45db757e520b84d42856529ae2f6fbdb9 WatchSource:0}: Error finding container 9cd0b5110ce17d9e4a03e094726497d45db757e520b84d42856529ae2f6fbdb9: Status 404 returned error can't find the container with id 9cd0b5110ce17d9e4a03e094726497d45db757e520b84d42856529ae2f6fbdb9
Oct 30 01:27:03.888938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:03.888476    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"]
Oct 30 01:27:04.915403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:04.915365    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh" event={"ID":"1efd82de-6184-4bb7-b070-5595a62c1284","Type":"ContainerStarted","Data":"249f9ac4dab7457a4d05b8afd196ab0a8218c8990a90f636c912807754c1150a"}
Oct 30 01:27:04.915941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:04.915452    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh" event={"ID":"1efd82de-6184-4bb7-b070-5595a62c1284","Type":"ContainerStarted","Data":"9cd0b5110ce17d9e4a03e094726497d45db757e520b84d42856529ae2f6fbdb9"}
Oct 30 01:27:05.926269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:05.926234    5104 generic.go:356] "Generic (PLEG): container finished" podID="1efd82de-6184-4bb7-b070-5595a62c1284" containerID="249f9ac4dab7457a4d05b8afd196ab0a8218c8990a90f636c912807754c1150a" exitCode=0
Oct 30 01:27:05.926721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:05.926314    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh" event={"ID":"1efd82de-6184-4bb7-b070-5595a62c1284","Type":"ContainerDied","Data":"249f9ac4dab7457a4d05b8afd196ab0a8218c8990a90f636c912807754c1150a"}
Oct 30 01:27:06.133939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.133911    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:06.175393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.174830    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efd82de-6184-4bb7-b070-5595a62c1284-kube-api-access-5v8wn\" (UniqueName: \"kubernetes.io/projected/1efd82de-6184-4bb7-b070-5595a62c1284-kube-api-access-5v8wn\") pod \"1efd82de-6184-4bb7-b070-5595a62c1284\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") "
Oct 30 01:27:06.175393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.175030    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-xks-ca\") pod \"1efd82de-6184-4bb7-b070-5595a62c1284\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") "
Oct 30 01:27:06.175393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.175069    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-openshift-ca\") pod \"1efd82de-6184-4bb7-b070-5595a62c1284\" (UID: \"1efd82de-6184-4bb7-b070-5595a62c1284\") "
Oct 30 01:27:06.178055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.177955    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-openshift-ca" pod "1efd82de-6184-4bb7-b070-5595a62c1284" (UID: "1efd82de-6184-4bb7-b070-5595a62c1284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:06.178968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.178937    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efd82de-6184-4bb7-b070-5595a62c1284-kube-api-access-5v8wn" pod "1efd82de-6184-4bb7-b070-5595a62c1284" (UID: "1efd82de-6184-4bb7-b070-5595a62c1284"). InnerVolumeSpecName "kube-api-access-5v8wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:06.184328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.180673    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-xks-ca" pod "1efd82de-6184-4bb7-b070-5595a62c1284" (UID: "1efd82de-6184-4bb7-b070-5595a62c1284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:06.278117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.278073    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8wn\" (UniqueName: \"kubernetes.io/projected/1efd82de-6184-4bb7-b070-5595a62c1284-kube-api-access-5v8wn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:06.278117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.278114    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:06.278349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:06.278129    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1efd82de-6184-4bb7-b070-5595a62c1284-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:07.049221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:07.049114    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"]
Oct 30 01:27:07.060510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:07.060436    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-eacea3-gvngs"]
Oct 30 01:27:07.286485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:07.272658    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh" event={"ID":"1efd82de-6184-4bb7-b070-5595a62c1284","Type":"ContainerDied","Data":"9cd0b5110ce17d9e4a03e094726497d45db757e520b84d42856529ae2f6fbdb9"}
Oct 30 01:27:07.286821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:07.286790    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cd0b5110ce17d9e4a03e094726497d45db757e520b84d42856529ae2f6fbdb9"
Oct 30 01:27:07.287030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:07.287016    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"
Oct 30 01:27:08.582166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:08.582125    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae5658b-b2dd-4684-a50f-52abed8430e3" path="/var/lib/kubelet/pods/7ae5658b-b2dd-4684-a50f-52abed8430e3/volumes"
Oct 30 01:27:08.904690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:08.904615    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:27:09.130598 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:27:09.130548    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:27:12.860731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:12.860663    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:12.902748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:12.902696    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:13.106630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:13.106566    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:13.686935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:13.686883    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:13.733323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:13.733272    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:27:14.107337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.107182    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"]
Oct 30 01:27:14.109840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.109812    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1efd82de-6184-4bb7-b070-5595a62c1284" containerName="revoke-keys"
Oct 30 01:27:14.110020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.109996    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efd82de-6184-4bb7-b070-5595a62c1284" containerName="revoke-keys"
Oct 30 01:27:14.110334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.110317    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efd82de-6184-4bb7-b070-5595a62c1284" containerName="revoke-keys"
Oct 30 01:27:14.135810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.122045    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.137845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.137803    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:27:14.170013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.169974    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"]
Oct 30 01:27:14.177426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.176584    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"]
Oct 30 01:27:14.177426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.176821    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" containerID="cri-o://82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f" gracePeriod=30
Oct 30 01:27:14.202940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.202899    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.203466 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.203420    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.203765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.203740    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppf6p\" (UniqueName: \"kubernetes.io/projected/eb858d6e-abbd-48a3-bb53-1050b07152a0-kube-api-access-ppf6p\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.250482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.246793    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:14.293483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.285080    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:27:14.319483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.293799    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:27:14.319839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.306872    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.320367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.320325    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.320704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.320682    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppf6p\" (UniqueName: \"kubernetes.io/projected/eb858d6e-abbd-48a3-bb53-1050b07152a0-kube-api-access-ppf6p\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.340478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.307714    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.380752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.380665    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppf6p\" (UniqueName: \"kubernetes.io/projected/eb858d6e-abbd-48a3-bb53-1050b07152a0-kube-api-access-ppf6p\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.459156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.459115    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:14.465850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:14.465817    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:15.361076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.361041    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"]
Oct 30 01:27:15.392394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.392355    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"]
Oct 30 01:27:15.392661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.392575    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.520494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.513785    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.520494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.514201    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xrx\" (UniqueName: \"kubernetes.io/projected/289b7280-d25a-47fc-8d56-5e2d873fc5fc-kube-api-access-w2xrx\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.520494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.514239    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.584639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.583039    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"]
Oct 30 01:27:15.584639 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:27:15.584085    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb858d6e_abbd_48a3_bb53_1050b07152a0.slice/crio-3640ee2752c2e5488066c2031ba7266b51b3318938bae5efe8397b3901d9b7a9 WatchSource:0}: Error finding container 3640ee2752c2e5488066c2031ba7266b51b3318938bae5efe8397b3901d9b7a9: Status 404 returned error can't find the container with id 3640ee2752c2e5488066c2031ba7266b51b3318938bae5efe8397b3901d9b7a9
Oct 30 01:27:15.597169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.596654    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"]
Oct 30 01:27:15.621924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.621879    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xrx\" (UniqueName: \"kubernetes.io/projected/289b7280-d25a-47fc-8d56-5e2d873fc5fc-kube-api-access-w2xrx\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.622135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.621951    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.622135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.622102    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.623295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.623262    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.624833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.624807    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.637396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.637361    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"]
Oct 30 01:27:15.645161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.645122    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xrx\" (UniqueName: \"kubernetes.io/projected/289b7280-d25a-47fc-8d56-5e2d873fc5fc-kube-api-access-w2xrx\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.719637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.715863    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:15.809899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:15.809853    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr" event={"ID":"eb858d6e-abbd-48a3-bb53-1050b07152a0","Type":"ContainerStarted","Data":"3640ee2752c2e5488066c2031ba7266b51b3318938bae5efe8397b3901d9b7a9"}
Oct 30 01:27:16.260849 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:27:16.260812    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289b7280_d25a_47fc_8d56_5e2d873fc5fc.slice/crio-3cd6d95ff9f90dc248428dfe9ebbf3186abc09eb60e43d47a06906d639a9fc23 WatchSource:0}: Error finding container 3cd6d95ff9f90dc248428dfe9ebbf3186abc09eb60e43d47a06906d639a9fc23: Status 404 returned error can't find the container with id 3cd6d95ff9f90dc248428dfe9ebbf3186abc09eb60e43d47a06906d639a9fc23
Oct 30 01:27:16.261008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:16.260971    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"]
Oct 30 01:27:16.823736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:16.823691    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr" event={"ID":"eb858d6e-abbd-48a3-bb53-1050b07152a0","Type":"ContainerStarted","Data":"9d78f75423346437967401e0edb140ed42b5fb3e2b91abb9952ca63a2b2a0e74"}
Oct 30 01:27:16.825984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:16.825948    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm" event={"ID":"289b7280-d25a-47fc-8d56-5e2d873fc5fc","Type":"ContainerStarted","Data":"3cd6d95ff9f90dc248428dfe9ebbf3186abc09eb60e43d47a06906d639a9fc23"}
Oct 30 01:27:16.866188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:16.866129    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr" podStartSLOduration=2.8661136369999998 podStartE2EDuration="2.866113637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:16.860373404 +0000 UTC m=+2678.408063900" watchObservedRunningTime="2026-10-30 01:27:16.866113637 +0000 UTC m=+2678.413804132"
Oct 30 01:27:17.837519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:17.837428    5104 generic.go:356] "Generic (PLEG): container finished" podID="eb858d6e-abbd-48a3-bb53-1050b07152a0" containerID="9d78f75423346437967401e0edb140ed42b5fb3e2b91abb9952ca63a2b2a0e74" exitCode=0
Oct 30 01:27:17.837519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:17.837492    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr" event={"ID":"eb858d6e-abbd-48a3-bb53-1050b07152a0","Type":"ContainerDied","Data":"9d78f75423346437967401e0edb140ed42b5fb3e2b91abb9952ca63a2b2a0e74"}
Oct 30 01:27:17.839531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:17.839502    5104 generic.go:356] "Generic (PLEG): container finished" podID="289b7280-d25a-47fc-8d56-5e2d873fc5fc" containerID="a112d3874a93c60573a240b2ec0fb9dc9c843c47dbf61cc1cc657f70a18c3925" exitCode=0
Oct 30 01:27:17.839674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:17.839583    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm" event={"ID":"289b7280-d25a-47fc-8d56-5e2d873fc5fc","Type":"ContainerDied","Data":"a112d3874a93c60573a240b2ec0fb9dc9c843c47dbf61cc1cc657f70a18c3925"}
Oct 30 01:27:19.071550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.071429    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:19.093660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.093622    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289b7280-d25a-47fc-8d56-5e2d873fc5fc-kube-api-access-w2xrx\" (UniqueName: \"kubernetes.io/projected/289b7280-d25a-47fc-8d56-5e2d873fc5fc-kube-api-access-w2xrx\") pod \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") "
Oct 30 01:27:19.093912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.093820    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-openshift-ca\") pod \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") "
Oct 30 01:27:19.094090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.094066    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-xks-ca\") pod \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\" (UID: \"289b7280-d25a-47fc-8d56-5e2d873fc5fc\") "
Oct 30 01:27:19.094713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.094664    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-openshift-ca" pod "289b7280-d25a-47fc-8d56-5e2d873fc5fc" (UID: "289b7280-d25a-47fc-8d56-5e2d873fc5fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:19.095182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.095158    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.097897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.097862    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-xks-ca" pod "289b7280-d25a-47fc-8d56-5e2d873fc5fc" (UID: "289b7280-d25a-47fc-8d56-5e2d873fc5fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:19.098170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.098147    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289b7280-d25a-47fc-8d56-5e2d873fc5fc-kube-api-access-w2xrx" pod "289b7280-d25a-47fc-8d56-5e2d873fc5fc" (UID: "289b7280-d25a-47fc-8d56-5e2d873fc5fc"). InnerVolumeSpecName "kube-api-access-w2xrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:19.139584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.139556    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:19.202595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.197564    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xrx\" (UniqueName: \"kubernetes.io/projected/289b7280-d25a-47fc-8d56-5e2d873fc5fc-kube-api-access-w2xrx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.202595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.197609    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289b7280-d25a-47fc-8d56-5e2d873fc5fc-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.300123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.300055    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-openshift-ca\") pod \"eb858d6e-abbd-48a3-bb53-1050b07152a0\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") "
Oct 30 01:27:19.300432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.300403    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-xks-ca\") pod \"eb858d6e-abbd-48a3-bb53-1050b07152a0\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") "
Oct 30 01:27:19.301231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.301114    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb858d6e-abbd-48a3-bb53-1050b07152a0-kube-api-access-ppf6p\" (UniqueName: \"kubernetes.io/projected/eb858d6e-abbd-48a3-bb53-1050b07152a0-kube-api-access-ppf6p\") pod \"eb858d6e-abbd-48a3-bb53-1050b07152a0\" (UID: \"eb858d6e-abbd-48a3-bb53-1050b07152a0\") "
Oct 30 01:27:19.305933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.305028    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-openshift-ca" pod "eb858d6e-abbd-48a3-bb53-1050b07152a0" (UID: "eb858d6e-abbd-48a3-bb53-1050b07152a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:19.313025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.312966    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb858d6e-abbd-48a3-bb53-1050b07152a0-kube-api-access-ppf6p" pod "eb858d6e-abbd-48a3-bb53-1050b07152a0" (UID: "eb858d6e-abbd-48a3-bb53-1050b07152a0"). InnerVolumeSpecName "kube-api-access-ppf6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:19.315585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.315554    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-xks-ca" pod "eb858d6e-abbd-48a3-bb53-1050b07152a0" (UID: "eb858d6e-abbd-48a3-bb53-1050b07152a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:19.405838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.405805    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppf6p\" (UniqueName: \"kubernetes.io/projected/eb858d6e-abbd-48a3-bb53-1050b07152a0-kube-api-access-ppf6p\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.405838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.405833    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.406064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.405847    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb858d6e-abbd-48a3-bb53-1050b07152a0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.502654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.502619    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:27:19.571825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.571787    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"]
Oct 30 01:27:19.573344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573313    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy"
Oct 30 01:27:19.573344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573343    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy"
Oct 30 01:27:19.573696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573389    5104 cpu_manager.go:414] "containerMap: removing container" podUID="eb858d6e-abbd-48a3-bb53-1050b07152a0" containerName="revoke-keys"
Oct 30 01:27:19.573696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573398    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb858d6e-abbd-48a3-bb53-1050b07152a0" containerName="revoke-keys"
Oct 30 01:27:19.573696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573470    5104 cpu_manager.go:414] "containerMap: removing container" podUID="289b7280-d25a-47fc-8d56-5e2d873fc5fc" containerName="revoke-keys"
Oct 30 01:27:19.573696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573478    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289b7280-d25a-47fc-8d56-5e2d873fc5fc" containerName="revoke-keys"
Oct 30 01:27:19.573896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573735    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy"
Oct 30 01:27:19.573896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573756    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="289b7280-d25a-47fc-8d56-5e2d873fc5fc" containerName="revoke-keys"
Oct 30 01:27:19.573896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.573785    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb858d6e-abbd-48a3-bb53-1050b07152a0" containerName="revoke-keys"
Oct 30 01:27:19.578977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.578949    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.608592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.608544    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"]
Oct 30 01:27:19.609260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.609225    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-kube-api-access-zv6sc\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-kube-api-access-zv6sc\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.609421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.609322    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-podinfo\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.609421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.609376    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-token\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-token\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.609628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.609422    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-envoy\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.609628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.609619    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-certs\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.609760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.609696    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-credential-socket\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.609991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.609966    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-socket\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.610085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.610027    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istiod-ca-cert\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.610184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.610118    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-data\") pod \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\" (UID: \"c081e5d8-e6c1-4739-8a2f-a363e06b8e12\") "
Oct 30 01:27:19.610286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.610262    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-certs" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:19.610532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.610449    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-socket" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:19.610712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.610681    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-data" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:19.610988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.610967    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istiod-ca-cert" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:19.611091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.611071    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.611157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.611097    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.611157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.611112    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.611531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.611492    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-credential-socket" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:19.614316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.614258    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-token" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:19.614703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.614666    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-kube-api-access-zv6sc" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "kube-api-access-zv6sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:19.614998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.614967    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-podinfo" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:27:19.615683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.615659    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-envoy" pod "c081e5d8-e6c1-4739-8a2f-a363e06b8e12" (UID: "c081e5d8-e6c1-4739-8a2f-a363e06b8e12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:19.712865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.712799    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713113    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713190    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713385    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713428    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713487    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713620    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713681    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47333922-85a0-4b2f-ba9e-f4766827a5a9-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.713928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.713906    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbhz\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-kube-api-access-cdbhz\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.714138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.714117    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.714205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.714148    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6sc\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-kube-api-access-zv6sc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.714205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.714168    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.714205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.714184    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.714205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.714199    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.714520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.714215    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c081e5d8-e6c1-4739-8a2f-a363e06b8e12-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:19.816552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.816497    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817019    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817167    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817237    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47333922-85a0-4b2f-ba9e-f4766827a5a9-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817367    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbhz\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-kube-api-access-cdbhz\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817477    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817610    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817680    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.817873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.817800    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.818207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.818177    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.818423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.816934    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.818673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.818651    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.818842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.818824    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.819613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.819589    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47333922-85a0-4b2f-ba9e-f4766827a5a9-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.820715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.820689    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.821301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.821266    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.838291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.838253    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.838487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.838468    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbhz\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-kube-api-access-cdbhz\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.865198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.865126    5104 generic.go:356] "Generic (PLEG): container finished" podID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerID="82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f" exitCode=0
Oct 30 01:27:19.865396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.865200    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"
Oct 30 01:27:19.865396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.865299    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" event={"ID":"c081e5d8-e6c1-4739-8a2f-a363e06b8e12","Type":"ContainerDied","Data":"82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f"}
Oct 30 01:27:19.865396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.865354    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw" event={"ID":"c081e5d8-e6c1-4739-8a2f-a363e06b8e12","Type":"ContainerDied","Data":"90e9e2c464cbd53e4bf377c56af1a17f7c4e11290f4294a2ca3d8d3412fe6f23"}
Oct 30 01:27:19.865396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.865388    5104 scope.go:122] "RemoveContainer" containerID="82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f"
Oct 30 01:27:19.868782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.868752    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"
Oct 30 01:27:19.868922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.868813    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr" event={"ID":"eb858d6e-abbd-48a3-bb53-1050b07152a0","Type":"ContainerDied","Data":"3640ee2752c2e5488066c2031ba7266b51b3318938bae5efe8397b3901d9b7a9"}
Oct 30 01:27:19.868922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.868865    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3640ee2752c2e5488066c2031ba7266b51b3318938bae5efe8397b3901d9b7a9"
Oct 30 01:27:19.876165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.876142    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"
Oct 30 01:27:19.876314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.876170    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm" event={"ID":"289b7280-d25a-47fc-8d56-5e2d873fc5fc","Type":"ContainerDied","Data":"3cd6d95ff9f90dc248428dfe9ebbf3186abc09eb60e43d47a06906d639a9fc23"}
Oct 30 01:27:19.876314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.876211    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd6d95ff9f90dc248428dfe9ebbf3186abc09eb60e43d47a06906d639a9fc23"
Oct 30 01:27:19.894047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.894022    5104 scope.go:122] "RemoveContainer" containerID="818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0"
Oct 30 01:27:19.910564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.910539    5104 scope.go:122] "RemoveContainer" containerID="82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f"
Oct 30 01:27:19.910769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.910743    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:19.910980 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:27:19.910953    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f\": container with ID starting with 82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f not found: ID does not exist" containerID="82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f"
Oct 30 01:27:19.911038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.910995    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f"} err="failed to get container status \"82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f\": rpc error: code = NotFound desc = could not find container \"82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f\": container with ID starting with 82e4b35c6eaa2fea064baf88249ec9453c455829711ce26ba8351b8c4ae30a9f not found: ID does not exist"
Oct 30 01:27:19.911038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.911020    5104 scope.go:122] "RemoveContainer" containerID="818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0"
Oct 30 01:27:19.911360 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:27:19.911334    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0\": container with ID starting with 818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0 not found: ID does not exist" containerID="818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0"
Oct 30 01:27:19.911417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.911369    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0"} err="failed to get container status \"818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0\": rpc error: code = NotFound desc = could not find container \"818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0\": container with ID starting with 818179981d5c37c67c20622921614592439ba145631a02716508015a866fc9d0 not found: ID does not exist"
Oct 30 01:27:19.921773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.921737    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"]
Oct 30 01:27:19.927561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:19.927535    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fqncw"]
Oct 30 01:27:20.235906 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:27:20.235874    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47333922_85a0_4b2f_ba9e_f4766827a5a9.slice/crio-a6318593342b51f53b8288e6d86a3ab80056618d5e346fc376b60ced7e2204dc WatchSource:0}: Error finding container a6318593342b51f53b8288e6d86a3ab80056618d5e346fc376b60ced7e2204dc: Status 404 returned error can't find the container with id a6318593342b51f53b8288e6d86a3ab80056618d5e346fc376b60ced7e2204dc
Oct 30 01:27:20.236337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.236234    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"]
Oct 30 01:27:20.239326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.239292    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:20.239444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.239361    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:20.239444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.239390    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:20.584970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.584882    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" path="/var/lib/kubelet/pods/c081e5d8-e6c1-4739-8a2f-a363e06b8e12/volumes"
Oct 30 01:27:20.597996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.597953    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh" podUID="c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" containerName="payload-processing" containerID="cri-o://ac69a29c010e5b03b1c84176a7591f3e10f7e1b6e5b82583ae0aaba3ae0d42df" gracePeriod=25
Oct 30 01:27:20.638777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.638719    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt" podUID="b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" containerName="payload-pre-processing" containerID="cri-o://b1d573d4771f1dd25cae4641a9fdec70bd57787ac2dcb63957af72fca1637fbd" gracePeriod=25
Oct 30 01:27:20.927669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.927205    5104 generic.go:356] "Generic (PLEG): container finished" podID="b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" containerID="b1d573d4771f1dd25cae4641a9fdec70bd57787ac2dcb63957af72fca1637fbd" exitCode=0
Oct 30 01:27:20.927980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.927357    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt" event={"ID":"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd","Type":"ContainerDied","Data":"b1d573d4771f1dd25cae4641a9fdec70bd57787ac2dcb63957af72fca1637fbd"}
Oct 30 01:27:20.931153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.931100    5104 generic.go:356] "Generic (PLEG): container finished" podID="c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" containerID="ac69a29c010e5b03b1c84176a7591f3e10f7e1b6e5b82583ae0aaba3ae0d42df" exitCode=0
Oct 30 01:27:20.931652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.931617    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh" event={"ID":"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd","Type":"ContainerDied","Data":"ac69a29c010e5b03b1c84176a7591f3e10f7e1b6e5b82583ae0aaba3ae0d42df"}
Oct 30 01:27:20.934591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.934565    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" event={"ID":"47333922-85a0-4b2f-ba9e-f4766827a5a9","Type":"ContainerStarted","Data":"cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f"}
Oct 30 01:27:20.934787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.934769    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" event={"ID":"47333922-85a0-4b2f-ba9e-f4766827a5a9","Type":"ContainerStarted","Data":"a6318593342b51f53b8288e6d86a3ab80056618d5e346fc376b60ced7e2204dc"}
Oct 30 01:27:20.998063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:20.997541    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podStartSLOduration=207.997518987 podStartE2EDuration="3m27.997518987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:20.986928725 +0000 UTC m=+2682.534619223" watchObservedRunningTime="2026-10-30 01:27:20.997518987 +0000 UTC m=+2682.545209500"
Oct 30 01:27:21.327859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.327828    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:27:21.329340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.329316    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-plugins-config-volume\") pod \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") "
Oct 30 01:27:21.329573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.329378    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-kube-api-access-lqs24\" (UniqueName: \"kubernetes.io/projected/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-kube-api-access-lqs24\") pod \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\" (UID: \"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd\") "
Oct 30 01:27:21.329852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.329823    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-plugins-config-volume" pod "c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" (UID: "c1e05841-5a7e-42f7-a1c8-7bda50c33cdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:21.330191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.330162    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.333397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.333366    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-kube-api-access-lqs24" pod "c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" (UID: "c1e05841-5a7e-42f7-a1c8-7bda50c33cdd"). InnerVolumeSpecName "kube-api-access-lqs24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:21.431879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.431836    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqs24\" (UniqueName: \"kubernetes.io/projected/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd-kube-api-access-lqs24\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.467911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.467881    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:27:21.532521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.532442    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-kube-api-access-kbwwv\" (UniqueName: \"kubernetes.io/projected/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-kube-api-access-kbwwv\") pod \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") "
Oct 30 01:27:21.533003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.532973    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-plugins-config-volume\") pod \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\" (UID: \"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd\") "
Oct 30 01:27:21.533421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.533395    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-plugins-config-volume" pod "b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" (UID: "b7a51238-3b16-4e4a-b87b-a19ddf8af5dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:21.534089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.534060    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.535269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.535229    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-kube-api-access-kbwwv" pod "b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" (UID: "b7a51238-3b16-4e4a-b87b-a19ddf8af5dd"). InnerVolumeSpecName "kube-api-access-kbwwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:21.540533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.540505    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"]
Oct 30 01:27:21.541548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541524    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" containerName="payload-pre-processing"
Oct 30 01:27:21.541548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541546    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" containerName="payload-pre-processing"
Oct 30 01:27:21.541737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541590    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" containerName="payload-processing"
Oct 30 01:27:21.541737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541598    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" containerName="payload-processing"
Oct 30 01:27:21.541737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541650    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy"
Oct 30 01:27:21.541737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541658    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy"
Oct 30 01:27:21.541956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541904    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c081e5d8-e6c1-4739-8a2f-a363e06b8e12" containerName="istio-proxy"
Oct 30 01:27:21.541956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541918    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" containerName="payload-pre-processing"
Oct 30 01:27:21.541956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.541929    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" containerName="payload-processing"
Oct 30 01:27:21.546154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.546131    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.568113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.568085    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"]
Oct 30 01:27:21.635654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.635607    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.635971    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.636128    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.636193    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xkd\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-kube-api-access-n4xkd\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.636304    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.636437    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.636548    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.636637    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.636724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.636708    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.637257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.637235    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbwwv\" (UniqueName: \"kubernetes.io/projected/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd-kube-api-access-kbwwv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.739188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739090    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739257    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739330    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739390    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739433    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xkd\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-kube-api-access-n4xkd\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739557    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739607    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739612    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739742    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739801    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739872    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.739967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.739914    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.740179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.740148    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.740681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.740659    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.742048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.742028    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.742220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.742200    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.758869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.758847    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.759146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.759123    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xkd\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-kube-api-access-n4xkd\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.864333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.864297    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:21.913517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.913080    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:21.913517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.913250    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:21.913517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:21.913300    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:22.017360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.017324    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"
Oct 30 01:27:22.017974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.017917    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh" event={"ID":"c1e05841-5a7e-42f7-a1c8-7bda50c33cdd","Type":"ContainerDied","Data":"20c991adbdfc01820b8e76a33be91cbf3b681f81909c69cabf756e83666e45e0"}
Oct 30 01:27:22.018120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.018061    5104 scope.go:122] "RemoveContainer" containerID="ac69a29c010e5b03b1c84176a7591f3e10f7e1b6e5b82583ae0aaba3ae0d42df"
Oct 30 01:27:22.022719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.022148    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"
Oct 30 01:27:22.022719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.022352    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt" event={"ID":"b7a51238-3b16-4e4a-b87b-a19ddf8af5dd","Type":"ContainerDied","Data":"7b6389b5d59dedc0f41194e7101859c734d3d7b29f500d7ec27c3da66db885a8"}
Oct 30 01:27:22.060041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.046415    5104 scope.go:122] "RemoveContainer" containerID="b1d573d4771f1dd25cae4641a9fdec70bd57787ac2dcb63957af72fca1637fbd"
Oct 30 01:27:22.159159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.131142    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"]
Oct 30 01:27:22.159159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.142511    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.159159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.158281    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.159159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.158427    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.159159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.158541    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxjp\" (UniqueName: \"kubernetes.io/projected/383d53c0-b6a8-403f-a313-418ecbc04aaf-kube-api-access-tpxjp\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.190479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.180085    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:22.245539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.241561    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:27:22.298480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.297508    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.298480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.297660    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.298480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.297749    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxjp\" (UniqueName: \"kubernetes.io/projected/383d53c0-b6a8-403f-a313-418ecbc04aaf-kube-api-access-tpxjp\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.320562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.301372    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.360345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.321212    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.412476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.411195    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"]
Oct 30 01:27:22.497395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.426113    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxjp\" (UniqueName: \"kubernetes.io/projected/383d53c0-b6a8-403f-a313-418ecbc04aaf-kube-api-access-tpxjp\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.497395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.444655    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:22.842442 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:27:22.833588    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3afcfed_ab8a_494d_a375_d5b88cf74fed.slice/crio-ec8383e05b424286c2273dcdb2a6229d4d1f011b97dc0c378a23bdcaa3d33f55 WatchSource:0}: Error finding container ec8383e05b424286c2273dcdb2a6229d4d1f011b97dc0c378a23bdcaa3d33f55: Status 404 returned error can't find the container with id ec8383e05b424286c2273dcdb2a6229d4d1f011b97dc0c378a23bdcaa3d33f55
Oct 30 01:27:22.850475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.850195    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:22.850475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.850289    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:22.850475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.850334    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:22.945976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.945648    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:22.945976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.945728    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:22.947612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.947283    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-755f6c9c4b-49xsh"]
Oct 30 01:27:22.947612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.947337    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"]
Oct 30 01:27:22.947612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.947381    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"]
Oct 30 01:27:22.947612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.947404    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-7b4f867b95-q47wt"]
Oct 30 01:27:22.947612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:22.947430    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"]
Oct 30 01:27:23.040637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:23.040596    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" event={"ID":"a3afcfed-ab8a-494d-a375-d5b88cf74fed","Type":"ContainerStarted","Data":"ec8383e05b424286c2273dcdb2a6229d4d1f011b97dc0c378a23bdcaa3d33f55"}
Oct 30 01:27:23.224866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:23.224283    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"]
Oct 30 01:27:23.945130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:23.943322    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:23.945130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:23.943389    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:24.127077 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.127034    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm" event={"ID":"383d53c0-b6a8-403f-a313-418ecbc04aaf","Type":"ContainerStarted","Data":"859edc91c96a568df21e33586d9ef9333871a666d0ce1f3408dacc4778387c95"}
Oct 30 01:27:24.131402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.131366    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" event={"ID":"a3afcfed-ab8a-494d-a375-d5b88cf74fed","Type":"ContainerStarted","Data":"cc1ef63d8cbf4ba76e887d328e825e3386a7e13a47d86a08d4b1e605c802ceb9"}
Oct 30 01:27:24.247668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.245955    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podStartSLOduration=211.245934471 podStartE2EDuration="3m31.245934471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:24.194033177 +0000 UTC m=+2685.741723674" watchObservedRunningTime="2026-10-30 01:27:24.245934471 +0000 UTC m=+2685.793624967"
Oct 30 01:27:24.463002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.462620    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"]
Oct 30 01:27:24.463002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.462818    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph" podUID="b228613f-dded-4bb1-90e0-6f918e4f58f9" containerName="payload-pre-processing" containerID="cri-o://703ced038ee97f8f0dcd6050b0ef11cc7930f6d4c491089f31c85d12f6010bc7" gracePeriod=30
Oct 30 01:27:24.670928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.670896    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a51238-3b16-4e4a-b87b-a19ddf8af5dd" path="/var/lib/kubelet/pods/b7a51238-3b16-4e4a-b87b-a19ddf8af5dd/volumes"
Oct 30 01:27:24.671496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.671447    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e05841-5a7e-42f7-a1c8-7bda50c33cdd" path="/var/lib/kubelet/pods/c1e05841-5a7e-42f7-a1c8-7bda50c33cdd/volumes"
Oct 30 01:27:24.870275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.870192    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:24.870640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.870615    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:24.870845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:24.870815    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:25.310482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.233080    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:25.310482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.233147    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:25.521318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.521272    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"]
Oct 30 01:27:25.792485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.781377    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:25.816115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.816059    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:25.860667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.860605    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:25.891512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.887287    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 01:27:25.912054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.912002    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": context deadline exceeded"
Oct 30 01:27:25.961135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.961097    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:25.961581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:25.961529    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:26.081973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.081919    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:27:26.179482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.175559    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm" event={"ID":"383d53c0-b6a8-403f-a313-418ecbc04aaf","Type":"ContainerStarted","Data":"3c51d6c4f8517d5723699bca41b1b2df8e9f2e588a4acc7fefd26598e19ddddb"}
Oct 30 01:27:26.193899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.193854    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:26.285230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.285175    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:26.285594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.285563    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:26.364597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.335158    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:26.457494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.453897    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:26.486900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.486856    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:26.554866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.554817    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:27:26.659890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.659776    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct" podUID="615c0cab-adfe-4284-83a6-63b0dfaaf811" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.83:50052\" within 1s: context deadline exceeded"
Oct 30 01:27:26.673469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.670392    5104 generic.go:356] "Generic (PLEG): container finished" podID="b228613f-dded-4bb1-90e0-6f918e4f58f9" containerID="703ced038ee97f8f0dcd6050b0ef11cc7930f6d4c491089f31c85d12f6010bc7" exitCode=0
Oct 30 01:27:26.673469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.672451    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph" event={"ID":"b228613f-dded-4bb1-90e0-6f918e4f58f9","Type":"ContainerDied","Data":"703ced038ee97f8f0dcd6050b0ef11cc7930f6d4c491089f31c85d12f6010bc7"}
Oct 30 01:27:26.673469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.673042    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:26.861479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.861257    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:26.882516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.882475    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:26.953488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.953076    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:27.005982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:26.979737    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:27.010621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.010574    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:27.012436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.012402    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:27:27.065106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.065063    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:27.130626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.123188    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"]
Oct 30 01:27:27.130626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.123305    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:27:27.130626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.123544    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:27:27.130626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.123576    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be164-vq9ls"]
Oct 30 01:27:27.237591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.229002    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b228613f-dded-4bb1-90e0-6f918e4f58f9-kube-api-access-82h6c\" (UniqueName: \"kubernetes.io/projected/b228613f-dded-4bb1-90e0-6f918e4f58f9-kube-api-access-82h6c\") pod \"b228613f-dded-4bb1-90e0-6f918e4f58f9\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") "
Oct 30 01:27:27.237591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.229474    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b228613f-dded-4bb1-90e0-6f918e4f58f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228613f-dded-4bb1-90e0-6f918e4f58f9-plugins-config-volume\") pod \"b228613f-dded-4bb1-90e0-6f918e4f58f9\" (UID: \"b228613f-dded-4bb1-90e0-6f918e4f58f9\") "
Oct 30 01:27:27.295514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.284742    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b228613f-dded-4bb1-90e0-6f918e4f58f9-plugins-config-volume" pod "b228613f-dded-4bb1-90e0-6f918e4f58f9" (UID: "b228613f-dded-4bb1-90e0-6f918e4f58f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:27.295514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.285026    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228613f-dded-4bb1-90e0-6f918e4f58f9-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:27.340480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.340371    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:27.366478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.341020    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:27:27.415894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.404098    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b228613f-dded-4bb1-90e0-6f918e4f58f9-kube-api-access-82h6c" pod "b228613f-dded-4bb1-90e0-6f918e4f58f9" (UID: "b228613f-dded-4bb1-90e0-6f918e4f58f9"). InnerVolumeSpecName "kube-api-access-82h6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:27.517666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.517563    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph" podUID="b228613f-dded-4bb1-90e0-6f918e4f58f9" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.1.85:9004: i/o timeout"
Oct 30 01:27:27.645730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.588433    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 01:27:27.670798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.670750    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:27:27.794032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.683376    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82h6c\" (UniqueName: \"kubernetes.io/projected/b228613f-dded-4bb1-90e0-6f918e4f58f9-kube-api-access-82h6c\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:27.794277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.714572    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:27.823481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.723531    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:27:27.823481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.763865    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:27.823481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.789530    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:27.823481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.806079    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:27.823481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.806841    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:27:27.823481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.819145    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:27:27.870900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.870854    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 01:27:27.896925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:27.871661    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:28.047534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.042513    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:28.047534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.042578    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:28.235490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.204878    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:28.268125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.268091    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:28.350695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.350583    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:28.485724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.341876    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:27:28.501212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.315198    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Oct 30 01:27:28.710322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.397980    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:27:28.743144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.442774    5104 generic.go:356] "Generic (PLEG): container finished" podID="383d53c0-b6a8-403f-a313-418ecbc04aaf" containerID="3c51d6c4f8517d5723699bca41b1b2df8e9f2e588a4acc7fefd26598e19ddddb" exitCode=0
Oct 30 01:27:28.743144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.674556    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:28.888516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.832016    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:28.888516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.872376    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:28.888516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.872439    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:28.918682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.890514    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:29.027532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.896507    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:27:29.059031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.896543    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:27:29.130491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.896566    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 01:27:29.130491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.896587    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 01:27:29.130491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.897314    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:29.132086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:28.943892    5104 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused" start-of-body=
Oct 30 01:27:29.135595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.135533    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.114:15021/healthz/ready\": dial tcp 10.132.1.114:15021: connect: connection refused"
Oct 30 01:27:29.139599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.017669    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"
Oct 30 01:27:29.293543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.285915    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": context deadline exceeded"
Oct 30 01:27:29.452838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.452783    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:29.495670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.495613    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:27:29.594432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.594345    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018cedd4-61d0-463a-a89b-2362bf769ec0" path="/var/lib/kubelet/pods/018cedd4-61d0-463a-a89b-2362bf769ec0/volumes"
Oct 30 01:27:29.596009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.595976    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm" event={"ID":"383d53c0-b6a8-403f-a313-418ecbc04aaf","Type":"ContainerDied","Data":"3c51d6c4f8517d5723699bca41b1b2df8e9f2e588a4acc7fefd26598e19ddddb"}
Oct 30 01:27:29.630011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.629973    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"]
Oct 30 01:27:29.647395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.647362    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b228613f-dded-4bb1-90e0-6f918e4f58f9" containerName="payload-pre-processing"
Oct 30 01:27:29.647699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.647677    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b228613f-dded-4bb1-90e0-6f918e4f58f9" containerName="payload-pre-processing"
Oct 30 01:27:29.648343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.648325    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b228613f-dded-4bb1-90e0-6f918e4f58f9" containerName="payload-pre-processing"
Oct 30 01:27:29.731621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:29.731567    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:30.186098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.100063    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:27:30.476799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.476687    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:30.687359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.687278    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:30.763449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.670106    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5" podUID="c24729d4-d473-4c88-bf24-1a337be196ee" containerName="payload-processing" containerID="cri-o://fc64a3b2fa6f39ea8db351b84c741c978a517f3d3bb719bdee3ce1def5f47023" gracePeriod=25
Oct 30 01:27:30.956065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.955998    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f12096-c6a9-4cb3-9ec6-a276d59f039b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:30.957049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.957019    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhkmc\" (UniqueName: \"kubernetes.io/projected/64f12096-c6a9-4cb3-9ec6-a276d59f039b-kube-api-access-bhkmc\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:30.958932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.717834    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:30.960523 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.670914    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:27:30.961204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.717705    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"]
Oct 30 01:27:31.029614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.780865    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:31.029614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.670064    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 01:27:31.029614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:30.725872    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:31.043503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.041806    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eaf753-dockercfg-bcbtc\""
Oct 30 01:27:31.043503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.041926    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eaf753\""
Oct 30 01:27:31.053045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.052999    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:31.053351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.053306    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:31.345292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345195    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph" event={"ID":"b228613f-dded-4bb1-90e0-6f918e4f58f9","Type":"ContainerDied","Data":"065ee11c8a39b8f07ffcd5fd89ca37b947f042d3b935a441589138a465e6573c"}
Oct 30 01:27:31.345549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345320    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:31.345549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345388    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:27:31.345549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345468    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:27:31.345549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345488    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"]
Oct 30 01:27:31.345801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345623    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:27:31.345801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345664    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:27:31.345801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345695    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:27:31.345801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345726    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:27:31.345801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345755    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:27:31.345801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345787    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:27:31.346133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345820    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:27:31.346133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345850    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:27:31.346133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345880    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:27:31.346133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.345911    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:27:31.346344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.346230    5104 scope.go:122] "RemoveContainer" containerID="703ced038ee97f8f0dcd6050b0ef11cc7930f6d4c491089f31c85d12f6010bc7"
Oct 30 01:27:31.346570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.346536    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f12096-c6a9-4cb3-9ec6-a276d59f039b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:31.346846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.346828    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhkmc\" (UniqueName: \"kubernetes.io/projected/64f12096-c6a9-4cb3-9ec6-a276d59f039b-kube-api-access-bhkmc\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:31.357537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.357147    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f12096-c6a9-4cb3-9ec6-a276d59f039b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:31.359352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.359326    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:31.361820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.359801    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:31.365814 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:27:31.362801    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228613f_dded_4bb1_90e0_6f918e4f58f9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228613f_dded_4bb1_90e0_6f918e4f58f9.slice/crio-065ee11c8a39b8f07ffcd5fd89ca37b947f042d3b935a441589138a465e6573c\": RecentStats: unable to find data in memory cache]"
Oct 30 01:27:31.365814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.365131    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:27:31.367211 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:27:31.367178    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228613f_dded_4bb1_90e0_6f918e4f58f9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228613f_dded_4bb1_90e0_6f918e4f58f9.slice/crio-065ee11c8a39b8f07ffcd5fd89ca37b947f042d3b935a441589138a465e6573c\": RecentStats: unable to find data in memory cache]"
Oct 30 01:27:31.368213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.368184    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm" event={"ID":"383d53c0-b6a8-403f-a313-418ecbc04aaf","Type":"ContainerDied","Data":"859edc91c96a568df21e33586d9ef9333871a666d0ce1f3408dacc4778387c95"}
Oct 30 01:27:31.368400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.368380    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="859edc91c96a568df21e33586d9ef9333871a666d0ce1f3408dacc4778387c95"
Oct 30 01:27:31.371361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.371335    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhkmc\" (UniqueName: \"kubernetes.io/projected/64f12096-c6a9-4cb3-9ec6-a276d59f039b-kube-api-access-bhkmc\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:31.380601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.380569    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:27:31.380847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.380827    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"]
Oct 30 01:27:31.445494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.441825    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-7675ddb75-cl2ph"]
Oct 30 01:27:31.462084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.462002    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"]
Oct 30 01:27:31.621536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.582587    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-openshift-ca\") pod \"383d53c0-b6a8-403f-a313-418ecbc04aaf\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") "
Oct 30 01:27:31.621536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.582824    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-xks-ca\") pod \"383d53c0-b6a8-403f-a313-418ecbc04aaf\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") "
Oct 30 01:27:31.638485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.626507    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383d53c0-b6a8-403f-a313-418ecbc04aaf-kube-api-access-tpxjp\" (UniqueName: \"kubernetes.io/projected/383d53c0-b6a8-403f-a313-418ecbc04aaf-kube-api-access-tpxjp\") pod \"383d53c0-b6a8-403f-a313-418ecbc04aaf\" (UID: \"383d53c0-b6a8-403f-a313-418ecbc04aaf\") "
Oct 30 01:27:31.931759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.631966    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-openshift-ca" pod "383d53c0-b6a8-403f-a313-418ecbc04aaf" (UID: "383d53c0-b6a8-403f-a313-418ecbc04aaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:31.932322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.786577    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-xks-ca" pod "383d53c0-b6a8-403f-a313-418ecbc04aaf" (UID: "383d53c0-b6a8-403f-a313-418ecbc04aaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:31.939806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.930909    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383d53c0-b6a8-403f-a313-418ecbc04aaf-kube-api-access-tpxjp" pod "383d53c0-b6a8-403f-a313-418ecbc04aaf" (UID: "383d53c0-b6a8-403f-a313-418ecbc04aaf"). InnerVolumeSpecName "kube-api-access-tpxjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:31.993513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:31.975087    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:32.010565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.010525    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body=
Oct 30 01:27:32.010883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.010848    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused"
Oct 30 01:27:32.012667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.012644    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:32.014288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.014265    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383d53c0-b6a8-403f-a313-418ecbc04aaf-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:32.014436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.014422    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxjp\" (UniqueName: \"kubernetes.io/projected/383d53c0-b6a8-403f-a313-418ecbc04aaf-kube-api-access-tpxjp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:32.259862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.259738    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"]
Oct 30 01:27:32.271718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.271686    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0fed33-nmlb6"]
Oct 30 01:27:32.502221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.500557    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:32.597061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.597019    5104 generic.go:356] "Generic (PLEG): container finished" podID="c24729d4-d473-4c88-bf24-1a337be196ee" containerID="fc64a3b2fa6f39ea8db351b84c741c978a517f3d3bb719bdee3ce1def5f47023" exitCode=0
Oct 30 01:27:32.642946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.635417    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"
Oct 30 01:27:32.666026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.665984    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989267ea-10a8-4299-a24e-ee194aa3616e" path="/var/lib/kubelet/pods/989267ea-10a8-4299-a24e-ee194aa3616e/volumes"
Oct 30 01:27:32.666859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.666833    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b228613f-dded-4bb1-90e0-6f918e4f58f9" path="/var/lib/kubelet/pods/b228613f-dded-4bb1-90e0-6f918e4f58f9/volumes"
Oct 30 01:27:32.667705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:32.667674    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5" event={"ID":"c24729d4-d473-4c88-bf24-1a337be196ee","Type":"ContainerDied","Data":"fc64a3b2fa6f39ea8db351b84c741c978a517f3d3bb719bdee3ce1def5f47023"}
Oct 30 01:27:33.039271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.039231    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:27:33.054598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.053797    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24729d4-d473-4c88-bf24-1a337be196ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24729d4-d473-4c88-bf24-1a337be196ee-plugins-config-volume\") pod \"c24729d4-d473-4c88-bf24-1a337be196ee\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") "
Oct 30 01:27:33.054598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.054406    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24729d4-d473-4c88-bf24-1a337be196ee-plugins-config-volume" pod "c24729d4-d473-4c88-bf24-1a337be196ee" (UID: "c24729d4-d473-4c88-bf24-1a337be196ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:33.054866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.054240    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24729d4-d473-4c88-bf24-1a337be196ee-kube-api-access-kb4lw\" (UniqueName: \"kubernetes.io/projected/c24729d4-d473-4c88-bf24-1a337be196ee-kube-api-access-kb4lw\") pod \"c24729d4-d473-4c88-bf24-1a337be196ee\" (UID: \"c24729d4-d473-4c88-bf24-1a337be196ee\") "
Oct 30 01:27:33.055494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.055420    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24729d4-d473-4c88-bf24-1a337be196ee-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:33.057809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.057708    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24729d4-d473-4c88-bf24-1a337be196ee-kube-api-access-kb4lw" pod "c24729d4-d473-4c88-bf24-1a337be196ee" (UID: "c24729d4-d473-4c88-bf24-1a337be196ee"). InnerVolumeSpecName "kube-api-access-kb4lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:33.128279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.128237    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:33.136504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.130218    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:33.151424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.149884    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:27:33.157329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.157299    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4lw\" (UniqueName: \"kubernetes.io/projected/c24729d4-d473-4c88-bf24-1a337be196ee-kube-api-access-kb4lw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:33.265680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.265239    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"]
Oct 30 01:27:33.628769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.628721    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t" event={"ID":"64f12096-c6a9-4cb3-9ec6-a276d59f039b","Type":"ContainerStarted","Data":"1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02"}
Oct 30 01:27:33.629042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.629015    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t" event={"ID":"64f12096-c6a9-4cb3-9ec6-a276d59f039b","Type":"ContainerStarted","Data":"84048f8debe9933562ec63172c1cb90ad31969c08c7de85d794338f821fbb6d8"}
Oct 30 01:27:33.629227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.629213    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:33.631720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.631697    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"
Oct 30 01:27:33.631880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.631699    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5" event={"ID":"c24729d4-d473-4c88-bf24-1a337be196ee","Type":"ContainerDied","Data":"e8b0128f6f8bbc6ff8c9d97fd8c8b47dfc0bbfff8266e0821fc01bc0146ace30"}
Oct 30 01:27:33.632038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.632025    5104 scope.go:122] "RemoveContainer" containerID="fc64a3b2fa6f39ea8db351b84c741c978a517f3d3bb719bdee3ce1def5f47023"
Oct 30 01:27:33.689423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.673185    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t" podStartSLOduration=6.673165794 podStartE2EDuration="6.673165794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:33.667630841 +0000 UTC m=+2695.215321374" watchObservedRunningTime="2026-10-30 01:27:33.673165794 +0000 UTC m=+2695.220856291"
Oct 30 01:27:33.741767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.741725    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"]
Oct 30 01:27:33.748492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:33.748445    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73831d-75bf4f98d-2lkh5"]
Oct 30 01:27:34.582272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:34.582225    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24729d4-d473-4c88-bf24-1a337be196ee" path="/var/lib/kubelet/pods/c24729d4-d473-4c88-bf24-1a337be196ee/volumes"
Oct 30 01:27:36.035848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.035805    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"]
Oct 30 01:27:36.041697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.041659    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b93c072-f5hv8"]
Oct 30 01:27:36.361251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.361113    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h" podUID="4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" containerName="payload-pre-processing" containerID="cri-o://fd74b6126dd7351a030e70e22aefa72efcd38765b852259d744e5fbfaf6d031d" gracePeriod=25
Oct 30 01:27:36.635109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.635007    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4689b723-1fce-46a9-b587-3ed813fcde23" path="/var/lib/kubelet/pods/4689b723-1fce-46a9-b587-3ed813fcde23/volumes"
Oct 30 01:27:36.709843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.685653    5104 generic.go:356] "Generic (PLEG): container finished" podID="4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" containerID="fd74b6126dd7351a030e70e22aefa72efcd38765b852259d744e5fbfaf6d031d" exitCode=0
Oct 30 01:27:36.709843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.685714    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h" event={"ID":"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf","Type":"ContainerDied","Data":"fd74b6126dd7351a030e70e22aefa72efcd38765b852259d744e5fbfaf6d031d"}
Oct 30 01:27:36.825476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.824198    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:27:36.857529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.853001    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-plugins-config-volume\") pod \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") "
Oct 30 01:27:36.857529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.853091    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-kube-api-access-bklnk\" (UniqueName: \"kubernetes.io/projected/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-kube-api-access-bklnk\") pod \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\" (UID: \"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf\") "
Oct 30 01:27:36.857529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.853485    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-plugins-config-volume" pod "4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" (UID: "4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:36.857529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.853869    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:36.857529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.855979    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-kube-api-access-bklnk" pod "4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" (UID: "4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf"). InnerVolumeSpecName "kube-api-access-bklnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:36.957425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.957379    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"]
Oct 30 01:27:36.959949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.959915    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" containerName="payload-pre-processing"
Oct 30 01:27:36.959949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.959947    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" containerName="payload-pre-processing"
Oct 30 01:27:36.960174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.960010    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c24729d4-d473-4c88-bf24-1a337be196ee" containerName="payload-processing"
Oct 30 01:27:36.960174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.960019    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24729d4-d473-4c88-bf24-1a337be196ee" containerName="payload-processing"
Oct 30 01:27:36.960174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.960084    5104 cpu_manager.go:414] "containerMap: removing container" podUID="383d53c0-b6a8-403f-a313-418ecbc04aaf" containerName="revoke-keys"
Oct 30 01:27:36.960174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.960093    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383d53c0-b6a8-403f-a313-418ecbc04aaf" containerName="revoke-keys"
Oct 30 01:27:36.960706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.960680    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" containerName="payload-pre-processing"
Oct 30 01:27:36.960835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.960716    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24729d4-d473-4c88-bf24-1a337be196ee" containerName="payload-processing"
Oct 30 01:27:36.960835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.960747    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="383d53c0-b6a8-403f-a313-418ecbc04aaf" containerName="revoke-keys"
Oct 30 01:27:36.963719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.963680    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklnk\" (UniqueName: \"kubernetes.io/projected/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf-kube-api-access-bklnk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:36.969967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.969935    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:36.970356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:36.970329    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"]
Oct 30 01:27:37.064833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.064795    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-data\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.064861    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-certs\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.064996    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.065031    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-credential-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.065056    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-token\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.065277    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcscz\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-kube-api-access-jcscz\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.065359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-podinfo\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.065424    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-envoy\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.065617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.065478    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istiod-ca-cert\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.166932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.166849    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcscz\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-kube-api-access-jcscz\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.166932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.166912    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-podinfo\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.166958    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-envoy\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.166984    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istiod-ca-cert\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167156    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-data\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167207    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-certs\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167298    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167327    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-credential-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167362    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-token\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167660    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-data\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167797    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.167933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.167915    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-certs\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.168092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.168066    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istiod-ca-cert\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.168092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.168080    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-credential-socket\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.169501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.169446    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-envoy\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.169752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.169732    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-podinfo\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.186091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.186063    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcscz\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-kube-api-access-jcscz\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.186264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.186065    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-token\") pod \"e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") " pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.305513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.305442    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:37.632263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.632235    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"]
Oct 30 01:27:37.632419 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:27:37.632266    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1dc8cc_7b83_457f_a38a_f95d44e85dd4.slice/crio-e3cf4470ec2071c12beaf22517819529023b4cd29c25d795e7b86245431a67f9 WatchSource:0}: Error finding container e3cf4470ec2071c12beaf22517819529023b4cd29c25d795e7b86245431a67f9: Status 404 returned error can't find the container with id e3cf4470ec2071c12beaf22517819529023b4cd29c25d795e7b86245431a67f9
Oct 30 01:27:37.636150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.636103    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:37.636293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.636204    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:37.636293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.636254    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:37.704171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.704129    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" event={"ID":"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4","Type":"ContainerStarted","Data":"e3cf4470ec2071c12beaf22517819529023b4cd29c25d795e7b86245431a67f9"}
Oct 30 01:27:37.707261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.707234    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"
Oct 30 01:27:37.707423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.707237    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h" event={"ID":"4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf","Type":"ContainerDied","Data":"6d3bb68ae93ef79377f223e4867009b1d010b882225a393aa43c2fcfb840bd75"}
Oct 30 01:27:37.707423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.707384    5104 scope.go:122] "RemoveContainer" containerID="fd74b6126dd7351a030e70e22aefa72efcd38765b852259d744e5fbfaf6d031d"
Oct 30 01:27:37.761874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.761835    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"]
Oct 30 01:27:37.773965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:37.773924    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-69dc90-76cd767bf5-7fj4h"]
Oct 30 01:27:38.612534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:38.606975    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf" path="/var/lib/kubelet/pods/4f4ec475-03c8-4c29-a9c5-3a5e7b48fbbf/volumes"
Oct 30 01:27:38.737487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:38.736629    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" event={"ID":"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4","Type":"ContainerStarted","Data":"0ebefb938cd2095ff6dd55692d7a8b8a98fb399044ebd251050673185da4c976"}
Oct 30 01:27:38.796481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:38.795820    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podStartSLOduration=2.795797623 podStartE2EDuration="2.795797623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:38.792083143 +0000 UTC m=+2700.339773640" watchObservedRunningTime="2026-10-30 01:27:38.795797623 +0000 UTC m=+2700.343488120"
Oct 30 01:27:39.137948 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:27:39.137907    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:27:39.316102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.315684    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:39.316102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.315956    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 01:27:39.316102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.316005    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 01:27:39.635843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.635812    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:27:39.639591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.639564    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:27:39.647959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.647932    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:27:39.650729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.650708    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:27:39.754925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.754890    5104 generic.go:356] "Generic (PLEG): container finished" podID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerID="acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3" exitCode=0
Oct 30 01:27:39.755121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.754944    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" event={"ID":"a93ddf6d-66f1-470c-aa41-91ca7a4c9996","Type":"ContainerDied","Data":"acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3"}
Oct 30 01:27:39.755905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.755884    5104 scope.go:122] "RemoveContainer" containerID="acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3"
Oct 30 01:27:39.757972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.757939    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:39.758075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.758014    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:39.758075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:39.758056    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:27:40.306481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:40.305889    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 01:27:40.306481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:40.305956    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 01:27:40.767740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:40.767701    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" event={"ID":"a93ddf6d-66f1-470c-aa41-91ca7a4c9996","Type":"ContainerStarted","Data":"4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a"}
Oct 30 01:27:41.135650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:41.134409    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:27:41.135650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:41.134566    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 01:27:41.135650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:41.134611    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 01:27:41.342483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:41.334403    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 01:27:41.342483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:41.334537    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 01:27:41.449479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:41.447723    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:27:42.138004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:42.137965    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 01:27:42.138696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:42.138661    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 01:27:42.330585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:42.312579    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 01:27:42.330881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:42.330847    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 01:27:42.725482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:42.725138    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"]
Oct 30 01:27:42.756171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:42.756126    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"]
Oct 30 01:27:43.185478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:43.184059    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 01:27:43.185478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:43.184133    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 01:27:43.540555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:43.325756    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 01:27:43.540555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:43.325817    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 01:27:43.561554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:43.547526    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"]
Oct 30 01:27:43.804837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:43.723008    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b1c5e7-2r7w9"]
Oct 30 01:27:44.169781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.169727    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 01:27:44.170188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.170151    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 01:27:44.323924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.323815    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 01:27:44.323924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.323882    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 01:27:44.381748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.381699    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:44.451260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.446258    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:44.490648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.490600    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:44.590667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.590627    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e298f917-fac1-454e-92a1-15647183312c" path="/var/lib/kubelet/pods/e298f917-fac1-454e-92a1-15647183312c/volumes"
Oct 30 01:27:44.630803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.630757    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:44.670326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:44.670293    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:27:45.160488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:45.156373    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 01:27:45.160488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:45.156442    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 01:27:45.305914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:45.305871    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 01:27:45.306204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:45.306172    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 01:27:46.133809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:46.133766    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 01:27:46.134333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:46.133837    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 01:27:46.731528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:46.731490    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:46.735875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:46.735698    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:46.753495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:46.752194    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:27:47.037163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.037080    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"]
Oct 30 01:27:47.044614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.044586    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6ef162b-9clfr"]
Oct 30 01:27:47.133310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.133274    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body=
Oct 30 01:27:47.133552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.133343    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused"
Oct 30 01:27:47.725690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.725651    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4" podUID="a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" containerName="payload-processing" containerID="cri-o://5fd4979d6539f84f46c17414c2bc7ec0895c11c75c72b47add24062a2fe386fc" gracePeriod=25
Oct 30 01:27:47.752125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.752039    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" containerID="cri-o://258c290869ead76d3bd1fc989d92418e0b25668f5d8eefa75cd04731da9540a2" gracePeriod=45
Oct 30 01:27:47.756741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.756705    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct" podUID="615c0cab-adfe-4284-83a6-63b0dfaaf811" containerName="payload-pre-processing" containerID="cri-o://272e47c14528a86d755f779504f80cdc707d9796e16358f44bb75cedad0f0d12" gracePeriod=25
Oct 30 01:27:47.767211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:47.767179    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": dial tcp 10.132.1.95:8000: connect: connection refused"
Oct 30 01:27:48.460928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:48.460895    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:27:48.805512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:48.805473    5104 generic.go:356] "Generic (PLEG): container finished" podID="a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" containerID="5fd4979d6539f84f46c17414c2bc7ec0895c11c75c72b47add24062a2fe386fc" exitCode=0
Oct 30 01:27:49.042321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:48.948269    5104 generic.go:356] "Generic (PLEG): container finished" podID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerID="258c290869ead76d3bd1fc989d92418e0b25668f5d8eefa75cd04731da9540a2" exitCode=0
Oct 30 01:27:49.119943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.119906    5104 generic.go:356] "Generic (PLEG): container finished" podID="615c0cab-adfe-4284-83a6-63b0dfaaf811" containerID="272e47c14528a86d755f779504f80cdc707d9796e16358f44bb75cedad0f0d12" exitCode=0
Oct 30 01:27:49.129108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.129066    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5059a4-78ed-414e-93b1-37fd49b2bcfc" path="/var/lib/kubelet/pods/bb5059a4-78ed-414e-93b1-37fd49b2bcfc/volumes"
Oct 30 01:27:49.129784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.129747    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4" event={"ID":"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6","Type":"ContainerDied","Data":"5fd4979d6539f84f46c17414c2bc7ec0895c11c75c72b47add24062a2fe386fc"}
Oct 30 01:27:49.129934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.129821    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" event={"ID":"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5","Type":"ContainerDied","Data":"258c290869ead76d3bd1fc989d92418e0b25668f5d8eefa75cd04731da9540a2"}
Oct 30 01:27:49.129934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.129860    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"]
Oct 30 01:27:49.129934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.129895    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct" event={"ID":"615c0cab-adfe-4284-83a6-63b0dfaaf811","Type":"ContainerDied","Data":"272e47c14528a86d755f779504f80cdc707d9796e16358f44bb75cedad0f0d12"}
Oct 30 01:27:49.185318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.185254    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:27:49.185664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.185634    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:27:49.185903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.185876    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:49.187946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.187917    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-69dc90-8pnrm"]
Oct 30 01:27:49.193855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.193276    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:27:49.194253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.194231    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:27:49.198182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.198117    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:27:49.219524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.219451    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615c0cab-adfe-4284-83a6-63b0dfaaf811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615c0cab-adfe-4284-83a6-63b0dfaaf811-plugins-config-volume\") pod \"615c0cab-adfe-4284-83a6-63b0dfaaf811\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") "
Oct 30 01:27:49.219741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.219576    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-plugins-config-volume\") pod \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") "
Oct 30 01:27:49.219741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.219642    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-dshm\") pod \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") "
Oct 30 01:27:49.219741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.219681    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tmp-dir\") pod \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") "
Oct 30 01:27:49.219936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.219796    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615c0cab-adfe-4284-83a6-63b0dfaaf811-kube-api-access-pp4tk\" (UniqueName: \"kubernetes.io/projected/615c0cab-adfe-4284-83a6-63b0dfaaf811-kube-api-access-pp4tk\") pod \"615c0cab-adfe-4284-83a6-63b0dfaaf811\" (UID: \"615c0cab-adfe-4284-83a6-63b0dfaaf811\") "
Oct 30 01:27:49.219936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.219832    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-kube-api-access-4nvpl\" (UniqueName: \"kubernetes.io/projected/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-kube-api-access-4nvpl\") pod \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") "
Oct 30 01:27:49.219936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.219891    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tls-certs\" (UniqueName: \"kubernetes.io/secret/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tls-certs\") pod \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") "
Oct 30 01:27:49.220104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.220041    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-kube-api-access-9fhjf\" (UniqueName: \"kubernetes.io/projected/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-kube-api-access-9fhjf\") pod \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\" (UID: \"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6\") "
Oct 30 01:27:49.220104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.220066    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615c0cab-adfe-4284-83a6-63b0dfaaf811-plugins-config-volume" pod "615c0cab-adfe-4284-83a6-63b0dfaaf811" (UID: "615c0cab-adfe-4284-83a6-63b0dfaaf811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:49.220104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.220072    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-plugins-config-volume" pod "a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" (UID: "a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:49.220284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.220072    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-home\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-home\") pod \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") "
Oct 30 01:27:49.220284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.220223    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-model-cache\") pod \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\" (UID: \"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5\") "
Oct 30 01:27:49.220406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.220306    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-home" pod "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" (UID: "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:49.220488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.220394    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tmp-dir" pod "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" (UID: "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:49.221556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.221528    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615c0cab-adfe-4284-83a6-63b0dfaaf811-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.221726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.221712    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.221844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.221828    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.221952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.221940    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.222378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.222354    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-model-cache" pod "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" (UID: "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:49.224650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.224623    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-kube-api-access-9fhjf" pod "a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" (UID: "a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6"). InnerVolumeSpecName "kube-api-access-9fhjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:49.224823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.224802    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-kube-api-access-4nvpl" pod "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" (UID: "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5"). InnerVolumeSpecName "kube-api-access-4nvpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:49.225435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.225414    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-dshm" pod "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" (UID: "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:49.232731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.232701    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tls-certs" pod "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" (UID: "753685cf-6a7e-4ecd-8fbf-a4173cd8dea5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:49.237375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.237336    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615c0cab-adfe-4284-83a6-63b0dfaaf811-kube-api-access-pp4tk" pod "615c0cab-adfe-4284-83a6-63b0dfaaf811" (UID: "615c0cab-adfe-4284-83a6-63b0dfaaf811"). InnerVolumeSpecName "kube-api-access-pp4tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:49.323430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.323336    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fhjf\" (UniqueName: \"kubernetes.io/projected/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6-kube-api-access-9fhjf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.323676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.323645    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.323676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.323668    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.323676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.323677    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4tk\" (UniqueName: \"kubernetes.io/projected/615c0cab-adfe-4284-83a6-63b0dfaaf811-kube-api-access-pp4tk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.323946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.323688    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvpl\" (UniqueName: \"kubernetes.io/projected/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-kube-api-access-4nvpl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.323946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.323697    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:27:49.772257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.772205    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:49.772446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:49.772305    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:27:50.132553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.132451    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"
Oct 30 01:27:50.132553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.132450    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6" event={"ID":"753685cf-6a7e-4ecd-8fbf-a4173cd8dea5","Type":"ContainerDied","Data":"e14291f14bd4b0dbba06c7728731b750d28975d97979e87bde9be253078348fe"}
Oct 30 01:27:50.132553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.132548    5104 scope.go:122] "RemoveContainer" containerID="258c290869ead76d3bd1fc989d92418e0b25668f5d8eefa75cd04731da9540a2"
Oct 30 01:27:50.134367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.134346    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"
Oct 30 01:27:50.134615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.134396    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct" event={"ID":"615c0cab-adfe-4284-83a6-63b0dfaaf811","Type":"ContainerDied","Data":"bba0a790ea545b661330b780ce69ddb1d345f5be4a7f2f18559c84c762ce37bb"}
Oct 30 01:27:50.138515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.136742    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"
Oct 30 01:27:50.138515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.136779    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4" event={"ID":"a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6","Type":"ContainerDied","Data":"16132d0453fc1dc28193505795566474208f86b79ecc3f1114ef531cb6188d38"}
Oct 30 01:27:50.139727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.139704    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:27:50.147820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.147798    5104 scope.go:122] "RemoveContainer" containerID="272e47c14528a86d755f779504f80cdc707d9796e16358f44bb75cedad0f0d12"
Oct 30 01:27:50.160353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.160326    5104 scope.go:122] "RemoveContainer" containerID="5fd4979d6539f84f46c17414c2bc7ec0895c11c75c72b47add24062a2fe386fc"
Oct 30 01:27:50.253943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.253904    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"]
Oct 30 01:27:50.261788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.261752    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4be164/ipp-route-4be164-kserve-85d9dfbd5-mtdd6"]
Oct 30 01:27:50.293897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.293860    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"]
Oct 30 01:27:50.303699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.303660    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3ed60-55b9dc7ff-zzxct"]
Oct 30 01:27:50.345487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.344527    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"]
Oct 30 01:27:50.345487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.345005    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3ed60-577c9b6d4d-dbpp4"]
Oct 30 01:27:50.584898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.584862    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289b7280-d25a-47fc-8d56-5e2d873fc5fc" path="/var/lib/kubelet/pods/289b7280-d25a-47fc-8d56-5e2d873fc5fc/volumes"
Oct 30 01:27:50.585530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.585510    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615c0cab-adfe-4284-83a6-63b0dfaaf811" path="/var/lib/kubelet/pods/615c0cab-adfe-4284-83a6-63b0dfaaf811/volumes"
Oct 30 01:27:50.586073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.586056    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" path="/var/lib/kubelet/pods/753685cf-6a7e-4ecd-8fbf-a4173cd8dea5/volumes"
Oct 30 01:27:50.586817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:50.586798    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" path="/var/lib/kubelet/pods/a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6/volumes"
Oct 30 01:27:55.313747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:55.313704    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"]
Oct 30 01:27:55.314251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:55.313963    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy" containerID="cri-o://9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6" gracePeriod=30
Oct 30 01:27:56.008699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.008658    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"]
Oct 30 01:27:56.009634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009614    5104 cpu_manager.go:414] "containerMap: removing container" podUID="615c0cab-adfe-4284-83a6-63b0dfaaf811" containerName="payload-pre-processing"
Oct 30 01:27:56.009713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009635    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615c0cab-adfe-4284-83a6-63b0dfaaf811" containerName="payload-pre-processing"
Oct 30 01:27:56.009713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009685    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" containerName="payload-processing"
Oct 30 01:27:56.009713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009691    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" containerName="payload-processing"
Oct 30 01:27:56.009822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009726    5104 cpu_manager.go:414] "containerMap: removing container" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main"
Oct 30 01:27:56.009822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009733    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main"
Oct 30 01:27:56.009898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009888    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5aecba7-e0b4-47e5-8258-d8b6e88bb5c6" containerName="payload-processing"
Oct 30 01:27:56.009935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009902    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="753685cf-6a7e-4ecd-8fbf-a4173cd8dea5" containerName="main"
Oct 30 01:27:56.009935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.009913    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="615c0cab-adfe-4284-83a6-63b0dfaaf811" containerName="payload-pre-processing"
Oct 30 01:27:56.016325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.016299    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.032990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.032960    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:27:56.040057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.040031    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:56.063504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.063430    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"]
Oct 30 01:27:56.152919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.152874    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxf2\" (UniqueName: \"kubernetes.io/projected/99d13cee-ed78-493e-88a4-464bc231f1de-kube-api-access-nkxf2\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.153175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.153154    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.153311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.153293    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.255156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.255120    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.255360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.255229    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.255448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.255365    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxf2\" (UniqueName: \"kubernetes.io/projected/99d13cee-ed78-493e-88a4-464bc231f1de-kube-api-access-nkxf2\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.255988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.255950    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.257681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.257658    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.275376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.275262    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxf2\" (UniqueName: \"kubernetes.io/projected/99d13cee-ed78-493e-88a4-464bc231f1de-kube-api-access-nkxf2\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.334708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.334668    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:27:56.671986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:56.671952    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"]
Oct 30 01:27:56.672904 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:27:56.672865    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d13cee_ed78_493e_88a4_464bc231f1de.slice/crio-a5ba7529acb8f42c1616c15e224d49e32c5be947a94b710b3977a1c8fb84265e WatchSource:0}: Error finding container a5ba7529acb8f42c1616c15e224d49e32c5be947a94b710b3977a1c8fb84265e: Status 404 returned error can't find the container with id a5ba7529acb8f42c1616c15e224d49e32c5be947a94b710b3977a1c8fb84265e
Oct 30 01:27:57.290476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:57.290419    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l" event={"ID":"99d13cee-ed78-493e-88a4-464bc231f1de","Type":"ContainerStarted","Data":"a5ba7529acb8f42c1616c15e224d49e32c5be947a94b710b3977a1c8fb84265e"}
Oct 30 01:27:57.771087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:57.771042    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"]
Oct 30 01:27:57.792838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:57.790235    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a6b650-98srh"]
Oct 30 01:27:58.302815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:58.302779    5104 generic.go:356] "Generic (PLEG): container finished" podID="99d13cee-ed78-493e-88a4-464bc231f1de" containerID="01d748627b54ec0665ab150e6b4abb80ced7990709469e91361f9fee43c27ff2" exitCode=0
Oct 30 01:27:58.303040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:58.302872    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l" event={"ID":"99d13cee-ed78-493e-88a4-464bc231f1de","Type":"ContainerDied","Data":"01d748627b54ec0665ab150e6b4abb80ced7990709469e91361f9fee43c27ff2"}
Oct 30 01:27:58.584853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:27:58.584767    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efd82de-6184-4bb7-b070-5595a62c1284" path="/var/lib/kubelet/pods/1efd82de-6184-4bb7-b070-5595a62c1284/volumes"
Oct 30 01:28:00.261064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.261028    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:28:00.326291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.326247    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-openshift-ca\") pod \"99d13cee-ed78-493e-88a4-464bc231f1de\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") "
Oct 30 01:28:00.326541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.326358    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-xks-ca\") pod \"99d13cee-ed78-493e-88a4-464bc231f1de\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") "
Oct 30 01:28:00.326541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.326427    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d13cee-ed78-493e-88a4-464bc231f1de-kube-api-access-nkxf2\" (UniqueName: \"kubernetes.io/projected/99d13cee-ed78-493e-88a4-464bc231f1de-kube-api-access-nkxf2\") pod \"99d13cee-ed78-493e-88a4-464bc231f1de\" (UID: \"99d13cee-ed78-493e-88a4-464bc231f1de\") "
Oct 30 01:28:00.326763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.326728    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-openshift-ca" pod "99d13cee-ed78-493e-88a4-464bc231f1de" (UID: "99d13cee-ed78-493e-88a4-464bc231f1de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:00.327597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.327566    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.329741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.329709    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-xks-ca" pod "99d13cee-ed78-493e-88a4-464bc231f1de" (UID: "99d13cee-ed78-493e-88a4-464bc231f1de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:00.330107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.330079    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d13cee-ed78-493e-88a4-464bc231f1de-kube-api-access-nkxf2" pod "99d13cee-ed78-493e-88a4-464bc231f1de" (UID: "99d13cee-ed78-493e-88a4-464bc231f1de"). InnerVolumeSpecName "kube-api-access-nkxf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.429614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.429577    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d13cee-ed78-493e-88a4-464bc231f1de-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.429614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.429612    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkxf2\" (UniqueName: \"kubernetes.io/projected/99d13cee-ed78-493e-88a4-464bc231f1de-kube-api-access-nkxf2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.458230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.458187    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"]
Oct 30 01:28:00.463733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.463699    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73831d-scjsr"]
Oct 30 01:28:00.655891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.655846    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb858d6e-abbd-48a3-bb53-1050b07152a0" path="/var/lib/kubelet/pods/eb858d6e-abbd-48a3-bb53-1050b07152a0/volumes"
Oct 30 01:28:00.725930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.725896    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:28:00.773218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.772987    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-kube-api-access-mr5zw\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-kube-api-access-mr5zw\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773089    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-certs\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773129    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-socket\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773410    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-podinfo\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773471    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-token\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-token\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773590    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-data\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773664    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-credential-socket\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773704    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istiod-ca-cert\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773754    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-envoy\") pod \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\" (UID: \"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7\") "
Oct 30 01:28:00.773959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.773836    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-socket" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.774710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.774067    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-certs" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.774710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.774426    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-data" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.775147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.775117    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-credential-socket" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.775358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.775334    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.775766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.775366    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.775766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.775379    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.775997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.775954    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istiod-ca-cert" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:00.777419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.777380    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-podinfo" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:00.778547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.778511    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-kube-api-access-mr5zw" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "kube-api-access-mr5zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.779394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.779365    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-envoy" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.779933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.779912    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-token" pod "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" (UID: "a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.798183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.798143    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"]
Oct 30 01:28:00.823885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.823802    5104 generic.go:356] "Generic (PLEG): container finished" podID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerID="9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6" exitCode=0
Oct 30 01:28:00.824320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.824286    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" event={"ID":"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7","Type":"ContainerDied","Data":"9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6"}
Oct 30 01:28:00.824904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.824873    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj" event={"ID":"a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7","Type":"ContainerDied","Data":"d5e069a1cd6de4d4f3daa80d72af6503b8453003911017dd9b0bcd9779573efa"}
Oct 30 01:28:00.825056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.825041    5104 scope.go:122] "RemoveContainer" containerID="9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6"
Oct 30 01:28:00.832173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.825203    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"
Oct 30 01:28:00.833417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.833396    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"
Oct 30 01:28:00.833680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.833431    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l" event={"ID":"99d13cee-ed78-493e-88a4-464bc231f1de","Type":"ContainerDied","Data":"a5ba7529acb8f42c1616c15e224d49e32c5be947a94b710b3977a1c8fb84265e"}
Oct 30 01:28:00.833680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.833544    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ba7529acb8f42c1616c15e224d49e32c5be947a94b710b3977a1c8fb84265e"
Oct 30 01:28:00.847085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.847051    5104 scope.go:122] "RemoveContainer" containerID="c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2"
Oct 30 01:28:00.861945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.861916    5104 scope.go:122] "RemoveContainer" containerID="9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6"
Oct 30 01:28:00.862533 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:00.862499    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6\": container with ID starting with 9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6 not found: ID does not exist" containerID="9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6"
Oct 30 01:28:00.862658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.862547    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6"} err="failed to get container status \"9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6\": rpc error: code = NotFound desc = could not find container \"9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6\": container with ID starting with 9f73195e3f9488c68376b9f09bc345bef974e00cfbaec7363e6d4d2e199a29a6 not found: ID does not exist"
Oct 30 01:28:00.862658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.862576    5104 scope.go:122] "RemoveContainer" containerID="c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2"
Oct 30 01:28:00.863082 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:00.863050    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2\": container with ID starting with c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2 not found: ID does not exist" containerID="c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2"
Oct 30 01:28:00.863184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.863107    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2"} err="failed to get container status \"c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2\": rpc error: code = NotFound desc = could not find container \"c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2\": container with ID starting with c3c21f5edcf94ee126517bd2e5fb1a027774365f0c7ec7e6ae20e71bbaf11be2 not found: ID does not exist"
Oct 30 01:28:00.877620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.877591    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.877620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.877619    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.877822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.877636    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.877822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.877649    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.877822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.877664    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.877822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.877676    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr5zw\" (UniqueName: \"kubernetes.io/projected/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7-kube-api-access-mr5zw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.903101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.903068    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"]
Oct 30 01:28:00.981481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:00.971288    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-fp5hj"]
Oct 30 01:28:02.661939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:02.661898    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" path="/var/lib/kubelet/pods/a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7/volumes"
Oct 30 01:28:06.479390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.479350    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"]
Oct 30 01:28:06.480428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480406    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy"
Oct 30 01:28:06.480515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480429    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy"
Oct 30 01:28:06.480515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480487    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy"
Oct 30 01:28:06.480515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480496    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy"
Oct 30 01:28:06.480628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480542    5104 cpu_manager.go:414] "containerMap: removing container" podUID="99d13cee-ed78-493e-88a4-464bc231f1de" containerName="revoke-keys"
Oct 30 01:28:06.480628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480551    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d13cee-ed78-493e-88a4-464bc231f1de" containerName="revoke-keys"
Oct 30 01:28:06.480741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480727    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy"
Oct 30 01:28:06.480741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.480740    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d13cee-ed78-493e-88a4-464bc231f1de" containerName="revoke-keys"
Oct 30 01:28:06.484991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.484965    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.503000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.502956    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"]
Oct 30 01:28:06.513500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.507813    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:28:06.513500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.507895    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:06.554388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.554350    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.554388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.554407    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7hc\" (UniqueName: \"kubernetes.io/projected/18e8f361-3b6c-479b-ae99-e8769cae1c46-kube-api-access-zt7hc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.554715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.554583    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.656341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.656289    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.656536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.656404    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7hc\" (UniqueName: \"kubernetes.io/projected/18e8f361-3b6c-479b-ae99-e8769cae1c46-kube-api-access-zt7hc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.656643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.656587    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.657535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.657508    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.659004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.658986    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.681980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.681952    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7hc\" (UniqueName: \"kubernetes.io/projected/18e8f361-3b6c-479b-ae99-e8769cae1c46-kube-api-access-zt7hc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:06.802769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:06.802680    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:07.017867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.017813    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"]
Oct 30 01:28:07.048716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.048673    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"]
Oct 30 01:28:07.134296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.134272    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"]
Oct 30 01:28:07.186411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.185936    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm" event={"ID":"18e8f361-3b6c-479b-ae99-e8769cae1c46","Type":"ContainerStarted","Data":"9e5f36b1989f904819c4c91aba5a693c1c70e0c75db9b4257e68faea1b0a5657"}
Oct 30 01:28:07.515028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.514990    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"]
Oct 30 01:28:07.515577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.515265    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442" podUID="ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" containerName="maas-api" containerID="cri-o://75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c" gracePeriod=30
Oct 30 01:28:07.801378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.801347    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"]
Oct 30 01:28:07.802405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:07.802362    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy" containerID="cri-o://4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a" gracePeriod=30
Oct 30 01:28:08.087717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.086688    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:28:08.312644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.311100    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-kube-api-access-dmsgn\" (UniqueName: \"kubernetes.io/projected/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-kube-api-access-dmsgn\") pod \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") "
Oct 30 01:28:08.312644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.311196    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-openshift-service-ca\") pod \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") "
Oct 30 01:28:08.312644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.311309    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-metrics-certs\") pod \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") "
Oct 30 01:28:08.332731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.332694    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-maas-api-tls\") pod \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\" (UID: \"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0\") "
Oct 30 01:28:08.353674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.351690    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-openshift-service-ca" pod "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" (UID: "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:08.365826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.365717    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-kube-api-access-dmsgn" pod "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" (UID: "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0"). InnerVolumeSpecName "kube-api-access-dmsgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:08.369361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.368809    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm" event={"ID":"18e8f361-3b6c-479b-ae99-e8769cae1c46","Type":"ContainerStarted","Data":"4e73fd99447efaa5e200d1a14b47fa6fe6834cf1ea4af50687e6210e4bae02cb"}
Oct 30 01:28:08.370957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.370846    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-metrics-certs" pod "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" (UID: "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:08.371791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.371764    5104 generic.go:356] "Generic (PLEG): container finished" podID="ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" containerID="75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c" exitCode=0
Oct 30 01:28:08.372095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.372072    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442" event={"ID":"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0","Type":"ContainerDied","Data":"75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c"}
Oct 30 01:28:08.372253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.372238    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442" event={"ID":"ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0","Type":"ContainerDied","Data":"9e36943f547ffc4cac600fc9f8f6d6e8f984f2b0204d19133f4b073c780de6d4"}
Oct 30 01:28:08.372402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.372388    5104 scope.go:122] "RemoveContainer" containerID="75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c"
Oct 30 01:28:08.373717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.373027    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-maas-api-tls" pod "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" (UID: "ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:08.373717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.373156    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"
Oct 30 01:28:08.406678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.406438    5104 scope.go:122] "RemoveContainer" containerID="75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c"
Oct 30 01:28:08.407811 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:08.407277    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c\": container with ID starting with 75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c not found: ID does not exist" containerID="75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c"
Oct 30 01:28:08.407811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.407317    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c"} err="failed to get container status \"75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c\": rpc error: code = NotFound desc = could not find container \"75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c\": container with ID starting with 75cfe63c11c6fe0cade3aaaf92fe9deacea74b16123b827466d3402fde684f4c not found: ID does not exist"
Oct 30 01:28:08.413436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.413380    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm" podStartSLOduration=2.413360663 podStartE2EDuration="2.413360663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:08.407564579 +0000 UTC m=+2729.955255073" watchObservedRunningTime="2026-10-30 01:28:08.413360663 +0000 UTC m=+2729.961051161"
Oct 30 01:28:08.435199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.435159    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"]
Oct 30 01:28:08.436859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.436830    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" containerName="maas-api"
Oct 30 01:28:08.436859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.436857    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" containerName="maas-api"
Oct 30 01:28:08.437159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.437147    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23653a1-d8ae-47cf-bbf5-00ed5c3c68a7" containerName="istio-proxy"
Oct 30 01:28:08.437217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.437168    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" containerName="maas-api"
Oct 30 01:28:08.441912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.441888    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.447024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.446989    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"]
Oct 30 01:28:08.448151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.448131    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:08.448151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.448151    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-maas-api-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:08.448312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.448176    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmsgn\" (UniqueName: \"kubernetes.io/projected/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-kube-api-access-dmsgn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:08.448312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.448199    5104 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0-openshift-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:08.463421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.463383    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3ed60-6b58457599-4r442"]
Oct 30 01:28:08.472543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.472352    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"]
Oct 30 01:28:08.483582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.483549    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nss4l\""
Oct 30 01:28:08.486650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.486616    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eaf753-serving-cert\""
Oct 30 01:28:08.488695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.486983    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:28:08.497119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.497088    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"]
Oct 30 01:28:08.512243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.512132    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3ed60-4xrdm"]
Oct 30 01:28:08.550381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.550339    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62267d3-7fa0-436e-a6fb-bf197972e269-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.550906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.550588    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkzd\" (UniqueName: \"kubernetes.io/projected/d62267d3-7fa0-436e-a6fb-bf197972e269-kube-api-access-5lkzd\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.550906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.550616    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.550906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.550765    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-metrics-certs\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.584113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.584075    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383d53c0-b6a8-403f-a313-418ecbc04aaf" path="/var/lib/kubelet/pods/383d53c0-b6a8-403f-a313-418ecbc04aaf/volumes"
Oct 30 01:28:08.584742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.584726    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0" path="/var/lib/kubelet/pods/ce00dac2-a9a8-4dd6-99ae-f2f2ad00cea0/volumes"
Oct 30 01:28:08.652224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.652133    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-metrics-certs\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.652524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.652488    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62267d3-7fa0-436e-a6fb-bf197972e269-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.652766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.652741    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkzd\" (UniqueName: \"kubernetes.io/projected/d62267d3-7fa0-436e-a6fb-bf197972e269-kube-api-access-5lkzd\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.652909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.652776    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.653238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.653213    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62267d3-7fa0-436e-a6fb-bf197972e269-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.655106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.655080    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-metrics-certs\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.655598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.655582    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.671944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.671911    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkzd\" (UniqueName: \"kubernetes.io/projected/d62267d3-7fa0-436e-a6fb-bf197972e269-kube-api-access-5lkzd\") pod \"maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:08.789493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:08.783555    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:09.107717 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:09.107673    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:28:09.200036 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:09.197969    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62267d3_7fa0_436e_a6fb_bf197972e269.slice/crio-4034807fd07ee285eced014d6ecf830cae911b8355943ddd41277795988a2da1 WatchSource:0}: Error finding container 4034807fd07ee285eced014d6ecf830cae911b8355943ddd41277795988a2da1: Status 404 returned error can't find the container with id 4034807fd07ee285eced014d6ecf830cae911b8355943ddd41277795988a2da1
Oct 30 01:28:09.200036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.198033    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"]
Oct 30 01:28:09.200036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.199608    5104 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:28:09.386619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.386562    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s" event={"ID":"d62267d3-7fa0-436e-a6fb-bf197972e269","Type":"ContainerStarted","Data":"4034807fd07ee285eced014d6ecf830cae911b8355943ddd41277795988a2da1"}
Oct 30 01:28:09.388999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.388956    5104 generic.go:356] "Generic (PLEG): container finished" podID="18e8f361-3b6c-479b-ae99-e8769cae1c46" containerID="4e73fd99447efaa5e200d1a14b47fa6fe6834cf1ea4af50687e6210e4bae02cb" exitCode=0
Oct 30 01:28:09.389156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.388989    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm" event={"ID":"18e8f361-3b6c-479b-ae99-e8769cae1c46","Type":"ContainerDied","Data":"4e73fd99447efaa5e200d1a14b47fa6fe6834cf1ea4af50687e6210e4bae02cb"}
Oct 30 01:28:09.541492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.541375    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"]
Oct 30 01:28:09.551932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.551895    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.578026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.577514    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"]
Oct 30 01:28:09.681419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.681368    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.681645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.681623    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5r7g\" (UniqueName: \"kubernetes.io/projected/6b74a284-5196-49b9-b39a-ea75fe641f8d-kube-api-access-d5r7g\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.681848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.681829    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.818783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.790668    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.818783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.792123    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.818783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.792561    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.818783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.793239    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5r7g\" (UniqueName: \"kubernetes.io/projected/6b74a284-5196-49b9-b39a-ea75fe641f8d-kube-api-access-d5r7g\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.855482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.824645    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.855482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.837756    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5r7g\" (UniqueName: \"kubernetes.io/projected/6b74a284-5196-49b9-b39a-ea75fe641f8d-kube-api-access-d5r7g\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:09.902614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:09.889974    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:10.192714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.192655    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"]
Oct 30 01:28:10.199134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.199110    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.213204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.213103    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"]
Oct 30 01:28:10.230444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.230410    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"]
Oct 30 01:28:10.230669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.230574    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.236722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.236691    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"]
Oct 30 01:28:10.294397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.293776    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"]
Oct 30 01:28:10.332003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.331960    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.332130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.332072    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.332205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.332166    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.332331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.332311    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.332411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.332354    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmwj\" (UniqueName: \"kubernetes.io/projected/04824d47-54d7-40f4-a83c-f53fac403ef4-kube-api-access-snmwj\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.332750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.332725    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpb8f\" (UniqueName: \"kubernetes.io/projected/97ad2cf2-974b-4f97-b873-24901d81153a-kube-api-access-dpb8f\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.413216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.412685    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt" event={"ID":"6b74a284-5196-49b9-b39a-ea75fe641f8d","Type":"ContainerStarted","Data":"8a984ad29f5e87350b6c37803069456b16ce0abb470a165f8da5c9dcd20e752d"}
Oct 30 01:28:10.415721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.415688    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s" event={"ID":"d62267d3-7fa0-436e-a6fb-bf197972e269","Type":"ContainerStarted","Data":"039e32a6a5d40f1710f832cd421440b987a1792c49af3424b341cce0995dc5ce"}
Oct 30 01:28:10.415852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.415802    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:10.434542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.434501    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpb8f\" (UniqueName: \"kubernetes.io/projected/97ad2cf2-974b-4f97-b873-24901d81153a-kube-api-access-dpb8f\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.434716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.434654    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.434716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.434694    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.434815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.434731    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.434874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.434828    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.434874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.434859    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmwj\" (UniqueName: \"kubernetes.io/projected/04824d47-54d7-40f4-a83c-f53fac403ef4-kube-api-access-snmwj\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.437014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.436977    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.437495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.437472    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.438789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.438768    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.439157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.439127    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.459897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.459820    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpb8f\" (UniqueName: \"kubernetes.io/projected/97ad2cf2-974b-4f97-b873-24901d81153a-kube-api-access-dpb8f\") pod \"maas-api-revoke-sub-6a1c7da5-7jv5j\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.461162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.461132    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmwj\" (UniqueName: \"kubernetes.io/projected/04824d47-54d7-40f4-a83c-f53fac403ef4-kube-api-access-snmwj\") pod \"maas-api-revoke-sub-dc5159e6-289pm\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.461563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.461523    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s" podStartSLOduration=1.98854748 podStartE2EDuration="2.461509485s" totalImagesPullingTime="472.962005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:10.455169942 +0000 UTC m=+2732.002860439" watchObservedRunningTime="2026-10-30 01:28:10.461509485 +0000 UTC m=+2732.009199981"
Oct 30 01:28:10.554758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.554723    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:10.558808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.558773    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:10.755215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.755188    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:10.874787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.874417    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-openshift-ca\") pod \"18e8f361-3b6c-479b-ae99-e8769cae1c46\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") "
Oct 30 01:28:10.874787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.874736    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e8f361-3b6c-479b-ae99-e8769cae1c46-kube-api-access-zt7hc\" (UniqueName: \"kubernetes.io/projected/18e8f361-3b6c-479b-ae99-e8769cae1c46-kube-api-access-zt7hc\") pod \"18e8f361-3b6c-479b-ae99-e8769cae1c46\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") "
Oct 30 01:28:10.877798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.874909    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-xks-ca\") pod \"18e8f361-3b6c-479b-ae99-e8769cae1c46\" (UID: \"18e8f361-3b6c-479b-ae99-e8769cae1c46\") "
Oct 30 01:28:10.878517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.878492    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-openshift-ca" pod "18e8f361-3b6c-479b-ae99-e8769cae1c46" (UID: "18e8f361-3b6c-479b-ae99-e8769cae1c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:10.906118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.906058    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-xks-ca" pod "18e8f361-3b6c-479b-ae99-e8769cae1c46" (UID: "18e8f361-3b6c-479b-ae99-e8769cae1c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:10.906860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.906834    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e8f361-3b6c-479b-ae99-e8769cae1c46-kube-api-access-zt7hc" pod "18e8f361-3b6c-479b-ae99-e8769cae1c46" (UID: "18e8f361-3b6c-479b-ae99-e8769cae1c46"). InnerVolumeSpecName "kube-api-access-zt7hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:10.978204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.978108    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:10.978204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.978152    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt7hc\" (UniqueName: \"kubernetes.io/projected/18e8f361-3b6c-479b-ae99-e8769cae1c46-kube-api-access-zt7hc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:10.978204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:10.978167    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e8f361-3b6c-479b-ae99-e8769cae1c46-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:11.059224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.059185    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"]
Oct 30 01:28:11.061314 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:11.061281    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04824d47_54d7_40f4_a83c_f53fac403ef4.slice/crio-85a362b21d485127c3dafe68d5ace2be425fe64e27b697e57d252b0dd56acff4 WatchSource:0}: Error finding container 85a362b21d485127c3dafe68d5ace2be425fe64e27b697e57d252b0dd56acff4: Status 404 returned error can't find the container with id 85a362b21d485127c3dafe68d5ace2be425fe64e27b697e57d252b0dd56acff4
Oct 30 01:28:11.094281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.094248    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"]
Oct 30 01:28:11.427443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.427408    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm" event={"ID":"04824d47-54d7-40f4-a83c-f53fac403ef4","Type":"ContainerStarted","Data":"85a362b21d485127c3dafe68d5ace2be425fe64e27b697e57d252b0dd56acff4"}
Oct 30 01:28:11.429492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.429437    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt" event={"ID":"6b74a284-5196-49b9-b39a-ea75fe641f8d","Type":"ContainerStarted","Data":"a7775e58d15aee2d57c3363b92bb0077b7da51e5b5097198eeab05b382007522"}
Oct 30 01:28:11.431093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.431062    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"
Oct 30 01:28:11.431241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.431088    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm" event={"ID":"18e8f361-3b6c-479b-ae99-e8769cae1c46","Type":"ContainerDied","Data":"9e5f36b1989f904819c4c91aba5a693c1c70e0c75db9b4257e68faea1b0a5657"}
Oct 30 01:28:11.431241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.431136    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5f36b1989f904819c4c91aba5a693c1c70e0c75db9b4257e68faea1b0a5657"
Oct 30 01:28:11.432654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.432627    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j" event={"ID":"97ad2cf2-974b-4f97-b873-24901d81153a","Type":"ContainerStarted","Data":"ea8479b0aac6bb383377cebdb723656f915457623cace43d38efe1e25badc89d"}
Oct 30 01:28:11.471345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:11.471288    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt" podStartSLOduration=2.47127301 podStartE2EDuration="2.47127301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:11.466764706 +0000 UTC m=+2733.014455202" watchObservedRunningTime="2026-10-30 01:28:11.47127301 +0000 UTC m=+2733.018963506"
Oct 30 01:28:12.022534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.019571    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" podUID="69499e2a-f2b6-4d79-8132-ed3b23a62752" containerName="payload-processing" containerID="cri-o://7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a" gracePeriod=25
Oct 30 01:28:12.049128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.049066    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" podUID="439a5722-bfae-47e0-b389-c682c528531a" containerName="payload-pre-processing" containerID="cri-o://6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6" gracePeriod=25
Oct 30 01:28:12.207168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.207131    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"]
Oct 30 01:28:12.207373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.207351    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" containerID="cri-o://0ebefb938cd2095ff6dd55692d7a8b8a98fb399044ebd251050673185da4c976" gracePeriod=30
Oct 30 01:28:12.221475 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.221422    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"]
Oct 30 01:28:12.224088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.224032    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy" containerID="cri-o://501772db4abba02a697620530d64a7ddfe24df6290c45aa63b43581aa16e4e4e" gracePeriod=30
Oct 30 01:28:12.943290 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.943191    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"]
Oct 30 01:28:12.944876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.944845    5104 cpu_manager.go:414] "containerMap: removing container" podUID="18e8f361-3b6c-479b-ae99-e8769cae1c46" containerName="revoke-keys"
Oct 30 01:28:12.945024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.944878    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e8f361-3b6c-479b-ae99-e8769cae1c46" containerName="revoke-keys"
Oct 30 01:28:12.945150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.945135    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e8f361-3b6c-479b-ae99-e8769cae1c46" containerName="revoke-keys"
Oct 30 01:28:12.948667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.948635    5104 generic.go:356] "Generic (PLEG): container finished" podID="04824d47-54d7-40f4-a83c-f53fac403ef4" containerID="db20aca251cd0c17d0c4a0beeda68ba418773023a39f5e7dcf51c9134eb989c0" exitCode=0
Oct 30 01:28:12.951112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.951083    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm" event={"ID":"04824d47-54d7-40f4-a83c-f53fac403ef4","Type":"ContainerDied","Data":"db20aca251cd0c17d0c4a0beeda68ba418773023a39f5e7dcf51c9134eb989c0"}
Oct 30 01:28:12.951219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.951136    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"]
Oct 30 01:28:12.951260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.951232    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:12.952761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.952736    5104 generic.go:356] "Generic (PLEG): container finished" podID="6b74a284-5196-49b9-b39a-ea75fe641f8d" containerID="a7775e58d15aee2d57c3363b92bb0077b7da51e5b5097198eeab05b382007522" exitCode=0
Oct 30 01:28:12.952951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.952922    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt" event={"ID":"6b74a284-5196-49b9-b39a-ea75fe641f8d","Type":"ContainerDied","Data":"a7775e58d15aee2d57c3363b92bb0077b7da51e5b5097198eeab05b382007522"}
Oct 30 01:28:12.956336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.956302    5104 generic.go:356] "Generic (PLEG): container finished" podID="97ad2cf2-974b-4f97-b873-24901d81153a" containerID="be5d57cd7eb0be6a2797837630b677a442b9f52b29a428245035749670807319" exitCode=0
Oct 30 01:28:12.956466 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:12.956434    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j" event={"ID":"97ad2cf2-974b-4f97-b873-24901d81153a","Type":"ContainerDied","Data":"be5d57cd7eb0be6a2797837630b677a442b9f52b29a428245035749670807319"}
Oct 30 01:28:13.070324 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:13.069747    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93ddf6d_66f1_470c_aa41_91ca7a4c9996.slice/crio-conmon-4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93ddf6d_66f1_470c_aa41_91ca7a4c9996.slice/crio-4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:28:13.083341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.081030    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.083341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.081361    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.083341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.081423    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lghtb\" (UniqueName: \"kubernetes.io/projected/4fe794d5-954f-4279-a5bd-51607de68f0a-kube-api-access-lghtb\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.183159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.183118    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.183346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.183219    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lghtb\" (UniqueName: \"kubernetes.io/projected/4fe794d5-954f-4279-a5bd-51607de68f0a-kube-api-access-lghtb\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.183346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.183339    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.184098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.183988    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.186298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.186272    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:28:13.186528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.186503    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.202470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.202422    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lghtb\" (UniqueName: \"kubernetes.io/projected/4fe794d5-954f-4279-a5bd-51607de68f0a-kube-api-access-lghtb\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.284701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.284660    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-credential-socket\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.284908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.284816    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-envoy\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.284908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.284883    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-data\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.285021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.284906    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-certs\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.285021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.284971    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-podinfo\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.285131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.285035    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-credential-socket" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:13.285131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.285100    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-kube-api-access-j9wtb\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-kube-api-access-j9wtb\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.285242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.285163    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-socket\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.285292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.285262    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-certs" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:13.285352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.285314    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-token\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-token\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.285449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.285398    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istiod-ca-cert\") pod \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\" (UID: \"a93ddf6d-66f1-470c-aa41-91ca7a4c9996\") "
Oct 30 01:28:13.286245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.285998    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-socket" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:13.286488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.286375    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istiod-ca-cert" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:13.286488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.286435    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-data" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:13.287497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.287095    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.287497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.287130    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.287497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.287147    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.287497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.287162    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.287497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.287178    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.287497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.287336    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:13.290190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.290150    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-token" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:13.290427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.290404    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-envoy" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:13.290569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.290471    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-kube-api-access-j9wtb" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "kube-api-access-j9wtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:13.291583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.291558    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-podinfo" pod "a93ddf6d-66f1-470c-aa41-91ca7a4c9996" (UID: "a93ddf6d-66f1-470c-aa41-91ca7a4c9996"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:13.389827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.389686    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.389827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.389721    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.389827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.389734    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.389827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.389748    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wtb\" (UniqueName: \"kubernetes.io/projected/a93ddf6d-66f1-470c-aa41-91ca7a4c9996-kube-api-access-j9wtb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.612773 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:13.612734    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe794d5_954f_4279_a5bd_51607de68f0a.slice/crio-8a665599c6e6f40e4c33d73c34369456ce22403250e7e775f5af4acb4652264e WatchSource:0}: Error finding container 8a665599c6e6f40e4c33d73c34369456ce22403250e7e775f5af4acb4652264e: Status 404 returned error can't find the container with id 8a665599c6e6f40e4c33d73c34369456ce22403250e7e775f5af4acb4652264e
Oct 30 01:28:13.613299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.613272    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"]
Oct 30 01:28:13.967956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.967920    5104 generic.go:356] "Generic (PLEG): container finished" podID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerID="4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a" exitCode=0
Oct 30 01:28:13.968150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.967965    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"
Oct 30 01:28:13.968150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.967999    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" event={"ID":"a93ddf6d-66f1-470c-aa41-91ca7a4c9996","Type":"ContainerDied","Data":"4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a"}
Oct 30 01:28:13.968150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.968053    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w" event={"ID":"a93ddf6d-66f1-470c-aa41-91ca7a4c9996","Type":"ContainerDied","Data":"6a224fb4bec90c4be721edd2663b183ee794c50adff237900253eb734473ae95"}
Oct 30 01:28:13.968150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.968076    5104 scope.go:122] "RemoveContainer" containerID="4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a"
Oct 30 01:28:13.969936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.969905    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk" event={"ID":"4fe794d5-954f-4279-a5bd-51607de68f0a","Type":"ContainerStarted","Data":"8a665599c6e6f40e4c33d73c34369456ce22403250e7e775f5af4acb4652264e"}
Oct 30 01:28:13.988310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:13.988284    5104 scope.go:122] "RemoveContainer" containerID="acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3"
Oct 30 01:28:14.012906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.012878    5104 scope.go:122] "RemoveContainer" containerID="4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a"
Oct 30 01:28:14.013282 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:14.013253    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a\": container with ID starting with 4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a not found: ID does not exist" containerID="4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a"
Oct 30 01:28:14.013421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.013293    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a"} err="failed to get container status \"4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a\": rpc error: code = NotFound desc = could not find container \"4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a\": container with ID starting with 4a90116c183206f74011ecf65cf9004ca98b7700deb0f0e167bb9de08b96bd6a not found: ID does not exist"
Oct 30 01:28:14.013421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.013318    5104 scope.go:122] "RemoveContainer" containerID="acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3"
Oct 30 01:28:14.013737 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:14.013713    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3\": container with ID starting with acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3 not found: ID does not exist" containerID="acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3"
Oct 30 01:28:14.013781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.013752    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3"} err="failed to get container status \"acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3\": rpc error: code = NotFound desc = could not find container \"acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3\": container with ID starting with acd669bef1d5474a2d6dca70959a74dfc6f4bb90e6ed1ec2baf41cf08d02bfb3 not found: ID does not exist"
Oct 30 01:28:14.035058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.034761    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"]
Oct 30 01:28:14.043929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.043890    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-73831d-openshift-default-d94dc898c-jgf8w"]
Oct 30 01:28:14.380406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.380362    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:14.386688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.386662    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:14.426777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.426708    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:14.522132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.522096    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b74a284-5196-49b9-b39a-ea75fe641f8d-kube-api-access-d5r7g\" (UniqueName: \"kubernetes.io/projected/6b74a284-5196-49b9-b39a-ea75fe641f8d-kube-api-access-d5r7g\") pod \"6b74a284-5196-49b9-b39a-ea75fe641f8d\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") "
Oct 30 01:28:14.522380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.522230    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-xks-ca\") pod \"04824d47-54d7-40f4-a83c-f53fac403ef4\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") "
Oct 30 01:28:14.522380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.522303    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ad2cf2-974b-4f97-b873-24901d81153a-kube-api-access-dpb8f\" (UniqueName: \"kubernetes.io/projected/97ad2cf2-974b-4f97-b873-24901d81153a-kube-api-access-dpb8f\") pod \"97ad2cf2-974b-4f97-b873-24901d81153a\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") "
Oct 30 01:28:14.522380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.522363    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04824d47-54d7-40f4-a83c-f53fac403ef4-kube-api-access-snmwj\" (UniqueName: \"kubernetes.io/projected/04824d47-54d7-40f4-a83c-f53fac403ef4-kube-api-access-snmwj\") pod \"04824d47-54d7-40f4-a83c-f53fac403ef4\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") "
Oct 30 01:28:14.523617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.523590    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-xks-ca\") pod \"6b74a284-5196-49b9-b39a-ea75fe641f8d\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") "
Oct 30 01:28:14.523757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.523642    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-openshift-ca\") pod \"6b74a284-5196-49b9-b39a-ea75fe641f8d\" (UID: \"6b74a284-5196-49b9-b39a-ea75fe641f8d\") "
Oct 30 01:28:14.523757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.523725    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-openshift-ca\") pod \"97ad2cf2-974b-4f97-b873-24901d81153a\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") "
Oct 30 01:28:14.523915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.523852    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-openshift-ca\") pod \"04824d47-54d7-40f4-a83c-f53fac403ef4\" (UID: \"04824d47-54d7-40f4-a83c-f53fac403ef4\") "
Oct 30 01:28:14.524035 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.523923    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-xks-ca\") pod \"97ad2cf2-974b-4f97-b873-24901d81153a\" (UID: \"97ad2cf2-974b-4f97-b873-24901d81153a\") "
Oct 30 01:28:14.525410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.524373    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-openshift-ca" pod "97ad2cf2-974b-4f97-b873-24901d81153a" (UID: "97ad2cf2-974b-4f97-b873-24901d81153a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:14.525410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.525069    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.525831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.525773    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-openshift-ca" pod "04824d47-54d7-40f4-a83c-f53fac403ef4" (UID: "04824d47-54d7-40f4-a83c-f53fac403ef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:14.526107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.526072    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-openshift-ca" pod "6b74a284-5196-49b9-b39a-ea75fe641f8d" (UID: "6b74a284-5196-49b9-b39a-ea75fe641f8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:14.526783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.526758    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04824d47-54d7-40f4-a83c-f53fac403ef4-kube-api-access-snmwj" pod "04824d47-54d7-40f4-a83c-f53fac403ef4" (UID: "04824d47-54d7-40f4-a83c-f53fac403ef4"). InnerVolumeSpecName "kube-api-access-snmwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:14.526918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.526861    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-xks-ca" pod "97ad2cf2-974b-4f97-b873-24901d81153a" (UID: "97ad2cf2-974b-4f97-b873-24901d81153a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:14.528247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.528221    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-xks-ca" pod "6b74a284-5196-49b9-b39a-ea75fe641f8d" (UID: "6b74a284-5196-49b9-b39a-ea75fe641f8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:14.534682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.529996    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ad2cf2-974b-4f97-b873-24901d81153a-kube-api-access-dpb8f" pod "97ad2cf2-974b-4f97-b873-24901d81153a" (UID: "97ad2cf2-974b-4f97-b873-24901d81153a"). InnerVolumeSpecName "kube-api-access-dpb8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:14.538315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.538283    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-xks-ca" pod "04824d47-54d7-40f4-a83c-f53fac403ef4" (UID: "04824d47-54d7-40f4-a83c-f53fac403ef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:14.539432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.539408    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b74a284-5196-49b9-b39a-ea75fe641f8d-kube-api-access-d5r7g" pod "6b74a284-5196-49b9-b39a-ea75fe641f8d" (UID: "6b74a284-5196-49b9-b39a-ea75fe641f8d"). InnerVolumeSpecName "kube-api-access-d5r7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:14.603750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.603710    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" path="/var/lib/kubelet/pods/a93ddf6d-66f1-470c-aa41-91ca7a4c9996/volumes"
Oct 30 01:28:14.626577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626547    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5r7g\" (UniqueName: \"kubernetes.io/projected/6b74a284-5196-49b9-b39a-ea75fe641f8d-kube-api-access-d5r7g\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.626577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626576    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.626756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626586    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpb8f\" (UniqueName: \"kubernetes.io/projected/97ad2cf2-974b-4f97-b873-24901d81153a-kube-api-access-dpb8f\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.626756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626595    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snmwj\" (UniqueName: \"kubernetes.io/projected/04824d47-54d7-40f4-a83c-f53fac403ef4-kube-api-access-snmwj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.626756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626604    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.626756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626612    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b74a284-5196-49b9-b39a-ea75fe641f8d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.626756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626621    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824d47-54d7-40f4-a83c-f53fac403ef4-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.626756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.626629    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ad2cf2-974b-4f97-b873-24901d81153a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.981684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.981649    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"
Oct 30 01:28:14.981684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.981659    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt" event={"ID":"6b74a284-5196-49b9-b39a-ea75fe641f8d","Type":"ContainerDied","Data":"8a984ad29f5e87350b6c37803069456b16ce0abb470a165f8da5c9dcd20e752d"}
Oct 30 01:28:14.982009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.981721    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a984ad29f5e87350b6c37803069456b16ce0abb470a165f8da5c9dcd20e752d"
Oct 30 01:28:14.985534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.985507    5104 generic.go:356] "Generic (PLEG): container finished" podID="4fe794d5-954f-4279-a5bd-51607de68f0a" containerID="717f29e467c198132137e05719b6c81fb126ee023726e81e3c7a3d4cedd3a830" exitCode=0
Oct 30 01:28:14.985699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.985586    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk" event={"ID":"4fe794d5-954f-4279-a5bd-51607de68f0a","Type":"ContainerDied","Data":"717f29e467c198132137e05719b6c81fb126ee023726e81e3c7a3d4cedd3a830"}
Oct 30 01:28:14.987278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.987254    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"
Oct 30 01:28:14.987419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.987275    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j" event={"ID":"97ad2cf2-974b-4f97-b873-24901d81153a","Type":"ContainerDied","Data":"ea8479b0aac6bb383377cebdb723656f915457623cace43d38efe1e25badc89d"}
Oct 30 01:28:14.987419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.987321    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea8479b0aac6bb383377cebdb723656f915457623cace43d38efe1e25badc89d"
Oct 30 01:28:14.988939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.988917    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm" event={"ID":"04824d47-54d7-40f4-a83c-f53fac403ef4","Type":"ContainerDied","Data":"85a362b21d485127c3dafe68d5ace2be425fe64e27b697e57d252b0dd56acff4"}
Oct 30 01:28:14.989052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.988960    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a362b21d485127c3dafe68d5ace2be425fe64e27b697e57d252b0dd56acff4"
Oct 30 01:28:14.989052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:14.988964    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"
Oct 30 01:28:15.767239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:15.767207    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"]
Oct 30 01:28:15.787505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:15.778238    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" containerID="cri-o://9d1ba47679fd8dfe61396c0aeded0563b7db0d82ba3db2e437c6017fc85ce242" gracePeriod=30
Oct 30 01:28:16.022719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:15.990110    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" containerID="cri-o://f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95" gracePeriod=45
Oct 30 01:28:16.496496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.489604    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:28:16.685351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.685313    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 01:28:16.743177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.742957    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:28:16.743177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.743023    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:28:16.866247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.866203    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"]
Oct 30 01:28:16.916758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.916697    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"]
Oct 30 01:28:16.981480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.981433    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:16.983409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:16.983382    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:28:17.113280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113098    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-dshm\") pod \"85f44d4e-7208-498f-bced-9b20007068b9\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") "
Oct 30 01:28:17.113280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113179    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-home\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-home\") pod \"85f44d4e-7208-498f-bced-9b20007068b9\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") "
Oct 30 01:28:17.113640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113291    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f44d4e-7208-498f-bced-9b20007068b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/85f44d4e-7208-498f-bced-9b20007068b9-tls-certs\") pod \"85f44d4e-7208-498f-bced-9b20007068b9\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") "
Oct 30 01:28:17.113640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113323    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-tmp-dir\") pod \"85f44d4e-7208-498f-bced-9b20007068b9\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") "
Oct 30 01:28:17.113640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113356    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-xks-ca\") pod \"4fe794d5-954f-4279-a5bd-51607de68f0a\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") "
Oct 30 01:28:17.113640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113476    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-openshift-ca\") pod \"4fe794d5-954f-4279-a5bd-51607de68f0a\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") "
Oct 30 01:28:17.113640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113532    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-model-cache\") pod \"85f44d4e-7208-498f-bced-9b20007068b9\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") "
Oct 30 01:28:17.113640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113579    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe794d5-954f-4279-a5bd-51607de68f0a-kube-api-access-lghtb\" (UniqueName: \"kubernetes.io/projected/4fe794d5-954f-4279-a5bd-51607de68f0a-kube-api-access-lghtb\") pod \"4fe794d5-954f-4279-a5bd-51607de68f0a\" (UID: \"4fe794d5-954f-4279-a5bd-51607de68f0a\") "
Oct 30 01:28:17.114018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.113787    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f44d4e-7208-498f-bced-9b20007068b9-kube-api-access-2z7gc\" (UniqueName: \"kubernetes.io/projected/85f44d4e-7208-498f-bced-9b20007068b9-kube-api-access-2z7gc\") pod \"85f44d4e-7208-498f-bced-9b20007068b9\" (UID: \"85f44d4e-7208-498f-bced-9b20007068b9\") "
Oct 30 01:28:17.115107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.115080    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-tmp-dir" pod "85f44d4e-7208-498f-bced-9b20007068b9" (UID: "85f44d4e-7208-498f-bced-9b20007068b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:17.119941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.118756    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-home" pod "85f44d4e-7208-498f-bced-9b20007068b9" (UID: "85f44d4e-7208-498f-bced-9b20007068b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:17.119941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.119614    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-openshift-ca" pod "4fe794d5-954f-4279-a5bd-51607de68f0a" (UID: "4fe794d5-954f-4279-a5bd-51607de68f0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:17.122574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.122549    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-model-cache" pod "85f44d4e-7208-498f-bced-9b20007068b9" (UID: "85f44d4e-7208-498f-bced-9b20007068b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:17.123233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.123195    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-dshm" pod "85f44d4e-7208-498f-bced-9b20007068b9" (UID: "85f44d4e-7208-498f-bced-9b20007068b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:17.123585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.123500    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe794d5-954f-4279-a5bd-51607de68f0a-kube-api-access-lghtb" pod "4fe794d5-954f-4279-a5bd-51607de68f0a" (UID: "4fe794d5-954f-4279-a5bd-51607de68f0a"). InnerVolumeSpecName "kube-api-access-lghtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:17.124956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.124929    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f44d4e-7208-498f-bced-9b20007068b9-kube-api-access-2z7gc" pod "85f44d4e-7208-498f-bced-9b20007068b9" (UID: "85f44d4e-7208-498f-bced-9b20007068b9"). InnerVolumeSpecName "kube-api-access-2z7gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:17.125258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.125238    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f44d4e-7208-498f-bced-9b20007068b9-tls-certs" pod "85f44d4e-7208-498f-bced-9b20007068b9" (UID: "85f44d4e-7208-498f-bced-9b20007068b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:17.126525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.126446    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-xks-ca" pod "4fe794d5-954f-4279-a5bd-51607de68f0a" (UID: "4fe794d5-954f-4279-a5bd-51607de68f0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217287    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7gc\" (UniqueName: \"kubernetes.io/projected/85f44d4e-7208-498f-bced-9b20007068b9-kube-api-access-2z7gc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217329    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217344    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217357    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f44d4e-7208-498f-bced-9b20007068b9-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217372    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217386    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217399    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe794d5-954f-4279-a5bd-51607de68f0a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217411    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85f44d4e-7208-498f-bced-9b20007068b9-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.217471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.217425    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lghtb\" (UniqueName: \"kubernetes.io/projected/4fe794d5-954f-4279-a5bd-51607de68f0a-kube-api-access-lghtb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:17.550686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.550648    5104 generic.go:356] "Generic (PLEG): container finished" podID="85f44d4e-7208-498f-bced-9b20007068b9" containerID="f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95" exitCode=0
Oct 30 01:28:17.550878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.550698    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"
Oct 30 01:28:17.550878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.550762    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" event={"ID":"85f44d4e-7208-498f-bced-9b20007068b9","Type":"ContainerDied","Data":"f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95"}
Oct 30 01:28:17.550878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.550797    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc" event={"ID":"85f44d4e-7208-498f-bced-9b20007068b9","Type":"ContainerDied","Data":"5201bb9486c21f3271bc056f5c1dce9db2393bd3e51df6de91111fec4dcce313"}
Oct 30 01:28:17.550878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.550813    5104 scope.go:122] "RemoveContainer" containerID="f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95"
Oct 30 01:28:17.594506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.593996    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk" event={"ID":"4fe794d5-954f-4279-a5bd-51607de68f0a","Type":"ContainerDied","Data":"8a665599c6e6f40e4c33d73c34369456ce22403250e7e775f5af4acb4652264e"}
Oct 30 01:28:17.594506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.594082    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a665599c6e6f40e4c33d73c34369456ce22403250e7e775f5af4acb4652264e"
Oct 30 01:28:17.594506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.594331    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"
Oct 30 01:28:17.609655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.604080    5104 generic.go:356] "Generic (PLEG): container finished" podID="70785039-29f0-4fee-ae6e-3719b53420c0" containerID="501772db4abba02a697620530d64a7ddfe24df6290c45aa63b43581aa16e4e4e" exitCode=0
Oct 30 01:28:17.609655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.604237    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" event={"ID":"70785039-29f0-4fee-ae6e-3719b53420c0","Type":"ContainerDied","Data":"501772db4abba02a697620530d64a7ddfe24df6290c45aa63b43581aa16e4e4e"}
Oct 30 01:28:17.609655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.607307    5104 generic.go:356] "Generic (PLEG): container finished" podID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerID="0ebefb938cd2095ff6dd55692d7a8b8a98fb399044ebd251050673185da4c976" exitCode=0
Oct 30 01:28:17.609655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.607397    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" event={"ID":"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4","Type":"ContainerDied","Data":"0ebefb938cd2095ff6dd55692d7a8b8a98fb399044ebd251050673185da4c976"}
Oct 30 01:28:17.974470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.974369    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"]
Oct 30 01:28:17.981628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.981574    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4fe794d5-954f-4279-a5bd-51607de68f0a" containerName="revoke-keys"
Oct 30 01:28:17.981864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.981848    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe794d5-954f-4279-a5bd-51607de68f0a" containerName="revoke-keys"
Oct 30 01:28:17.981999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.981988    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy"
Oct 30 01:28:17.982084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982074    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy"
Oct 30 01:28:17.982223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982212    5104 cpu_manager.go:414] "containerMap: removing container" podUID="04824d47-54d7-40f4-a83c-f53fac403ef4" containerName="revoke-keys"
Oct 30 01:28:17.982307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982298    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04824d47-54d7-40f4-a83c-f53fac403ef4" containerName="revoke-keys"
Oct 30 01:28:17.982403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982394    5104 cpu_manager.go:414] "containerMap: removing container" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main"
Oct 30 01:28:17.982500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982489    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main"
Oct 30 01:28:17.982641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982631    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6b74a284-5196-49b9-b39a-ea75fe641f8d" containerName="revoke-keys"
Oct 30 01:28:17.982714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982705    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b74a284-5196-49b9-b39a-ea75fe641f8d" containerName="revoke-keys"
Oct 30 01:28:17.982878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982868    5104 cpu_manager.go:414] "containerMap: removing container" podUID="97ad2cf2-974b-4f97-b873-24901d81153a" containerName="revoke-keys"
Oct 30 01:28:17.982961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.982950    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ad2cf2-974b-4f97-b873-24901d81153a" containerName="revoke-keys"
Oct 30 01:28:17.983102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.983090    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy"
Oct 30 01:28:17.983183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.983173    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy"
Oct 30 01:28:17.985516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.985492    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ad2cf2-974b-4f97-b873-24901d81153a" containerName="revoke-keys"
Oct 30 01:28:17.985713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.985697    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy"
Oct 30 01:28:17.985838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.985828    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe794d5-954f-4279-a5bd-51607de68f0a" containerName="revoke-keys"
Oct 30 01:28:17.985926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.985916    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93ddf6d-66f1-470c-aa41-91ca7a4c9996" containerName="istio-proxy"
Oct 30 01:28:17.986030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.986020    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="04824d47-54d7-40f4-a83c-f53fac403ef4" containerName="revoke-keys"
Oct 30 01:28:17.986117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.986108    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f44d4e-7208-498f-bced-9b20007068b9" containerName="main"
Oct 30 01:28:17.986211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.986201    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b74a284-5196-49b9-b39a-ea75fe641f8d" containerName="revoke-keys"
Oct 30 01:28:17.993509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.993484    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:17.999539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:17.996710    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"]
Oct 30 01:28:18.011188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.010960    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:28:18.032937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.011952    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:18.032937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.030915    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"]
Oct 30 01:28:18.040013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.039160    5104 scope.go:122] "RemoveContainer" containerID="f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95"
Oct 30 01:28:18.041026 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:18.040821    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95\": container with ID starting with f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95 not found: ID does not exist" containerID="f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95"
Oct 30 01:28:18.041026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.040880    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95"} err="failed to get container status \"f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95\": rpc error: code = NotFound desc = could not find container \"f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95\": container with ID starting with f99e6cd0d44d0acf5c608085fcf9e14efdf2cbe1710d91ca8799493c115c3c95 not found: ID does not exist"
Oct 30 01:28:18.041026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.040910    5104 scope.go:122] "RemoveContainer" containerID="a70cc2a52a6d25ab3952f306807882193d04bac7026d931c0709e4c5531d8bec"
Oct 30 01:28:18.052900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.052473    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/premium-w0-ba3d7b-kserve-6598984495-gsnfc"]
Oct 30 01:28:18.211858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.163775    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.234525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.212233    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92n5\" (UniqueName: \"kubernetes.io/projected/8c5545cc-2265-474c-978f-cb70a1efa5aa-kube-api-access-k92n5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.234525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.212543    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.292941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.292904    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:28:18.297421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.297396    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:28:18.334247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334213    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70785039-29f0-4fee-ae6e-3719b53420c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70785039-29f0-4fee-ae6e-3719b53420c0-istiod-ca-cert\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.334426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334263    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-token\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.334426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334313    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-socket\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.334426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334339    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-credential-socket\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.334426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334370    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-istio-token\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-istio-token\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.334746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334429    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70785039-29f0-4fee-ae6e-3719b53420c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70785039-29f0-4fee-ae6e-3719b53420c0-istio-podinfo\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.334746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334489    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-data\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.334746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334520    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-kube-api-access-vffg7\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-kube-api-access-vffg7\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.334746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334593    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-certs\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.334746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334654    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-certs\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.334746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334713    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70785039-29f0-4fee-ae6e-3719b53420c0-istiod-ca-cert" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:18.334746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334731    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-credential-socket" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.335147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334828    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-data" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.335147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334723    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-podinfo\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.335147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.334953    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-socket\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.335147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335043    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-envoy\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.335147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335124    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-data\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.335433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335041    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-socket" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.335433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335067    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-certs" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.335433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335235    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-socket" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.335433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335310    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-envoy\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.335433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335369    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-credential-socket\") pod \"70785039-29f0-4fee-ae6e-3719b53420c0\" (UID: \"70785039-29f0-4fee-ae6e-3719b53420c0\") "
Oct 30 01:28:18.335433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335401    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istiod-ca-cert\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.335433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335410    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-certs" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.336068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335524    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-kube-api-access-jcscz\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-kube-api-access-jcscz\") pod \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\" (UID: \"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4\") "
Oct 30 01:28:18.336068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.335924    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92n5\" (UniqueName: \"kubernetes.io/projected/8c5545cc-2265-474c-978f-cb70a1efa5aa-kube-api-access-k92n5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.336214 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336194    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.336579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336557    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336788    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336811    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336824    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336836    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336849    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70785039-29f0-4fee-ae6e-3719b53420c0-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336865    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.336877    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.337251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.337142    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70785039-29f0-4fee-ae6e-3719b53420c0-istio-podinfo" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:18.337999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.337687    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-credential-socket" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.337999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.337748    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-data" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.337999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.337753    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.337999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.337811    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-envoy" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.337999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.337934    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-token" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:18.338269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.338038    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-podinfo" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:18.338269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.338204    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-envoy" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:18.338269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.338229    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-istio-token" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:18.338269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.338251    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istiod-ca-cert" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:18.339703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.339672    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-kube-api-access-vffg7" pod "70785039-29f0-4fee-ae6e-3719b53420c0" (UID: "70785039-29f0-4fee-ae6e-3719b53420c0"). InnerVolumeSpecName "kube-api-access-vffg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:18.339810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.339784    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-kube-api-access-jcscz" pod "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" (UID: "ba1dc8cc-7b83-457f-a38a-f95d44e85dd4"). InnerVolumeSpecName "kube-api-access-jcscz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:18.340873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.340842    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.366006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.365975    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92n5\" (UniqueName: \"kubernetes.io/projected/8c5545cc-2265-474c-978f-cb70a1efa5aa-kube-api-access-k92n5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.439120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439086    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439119    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439128    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70785039-29f0-4fee-ae6e-3719b53420c0-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439138    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vffg7\" (UniqueName: \"kubernetes.io/projected/70785039-29f0-4fee-ae6e-3719b53420c0-kube-api-access-vffg7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439148    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439156    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439164    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439171    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439179    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70785039-29f0-4fee-ae6e-3719b53420c0-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439187    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.439380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.439196    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcscz\" (UniqueName: \"kubernetes.io/projected/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4-kube-api-access-jcscz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:18.584554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.584441    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f44d4e-7208-498f-bced-9b20007068b9" path="/var/lib/kubelet/pods/85f44d4e-7208-498f-bced-9b20007068b9/volumes"
Oct 30 01:28:18.621550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.621515    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"
Oct 30 01:28:18.622332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.621499    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb" event={"ID":"70785039-29f0-4fee-ae6e-3719b53420c0","Type":"ContainerDied","Data":"a06f4bfebb027edeacdd6a54a6a2a4e56788169e405f63c5ad285c7db6e5d275"}
Oct 30 01:28:18.622332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.621804    5104 scope.go:122] "RemoveContainer" containerID="501772db4abba02a697620530d64a7ddfe24df6290c45aa63b43581aa16e4e4e"
Oct 30 01:28:18.623867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.623791    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz" event={"ID":"ba1dc8cc-7b83-457f-a38a-f95d44e85dd4","Type":"ContainerDied","Data":"e3cf4470ec2071c12beaf22517819529023b4cd29c25d795e7b86245431a67f9"}
Oct 30 01:28:18.624038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.624022    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"
Oct 30 01:28:18.636651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.636621    5104 scope.go:122] "RemoveContainer" containerID="0ebefb938cd2095ff6dd55692d7a8b8a98fb399044ebd251050673185da4c976"
Oct 30 01:28:18.645363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.645328    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:18.680014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.679950    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"]
Oct 30 01:28:18.689062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.689028    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-68vtz"]
Oct 30 01:28:18.719613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.719540    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"]
Oct 30 01:28:18.727802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.727770    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be164-openshift-default-7446b5bdf9-hrlqb"]
Oct 30 01:28:18.961021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:18.960975    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"]
Oct 30 01:28:18.961186 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:18.961128    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5545cc_2265_474c_978f_cb70a1efa5aa.slice/crio-e52541217bca0f1802a55143dbaefacfb1653cfea6eace49a3fdb379cbc41fcd WatchSource:0}: Error finding container e52541217bca0f1802a55143dbaefacfb1653cfea6eace49a3fdb379cbc41fcd: Status 404 returned error can't find the container with id e52541217bca0f1802a55143dbaefacfb1653cfea6eace49a3fdb379cbc41fcd
Oct 30 01:28:19.640409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:19.640312    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl" event={"ID":"8c5545cc-2265-474c-978f-cb70a1efa5aa","Type":"ContainerStarted","Data":"d1e2db4f7aa14f14a737031b2067258c17936d7bedbf91fc58e667ce720ab961"}
Oct 30 01:28:19.640409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:19.640369    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl" event={"ID":"8c5545cc-2265-474c-978f-cb70a1efa5aa","Type":"ContainerStarted","Data":"e52541217bca0f1802a55143dbaefacfb1653cfea6eace49a3fdb379cbc41fcd"}
Oct 30 01:28:19.676909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:19.676844    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl" podStartSLOduration=2.676823227 podStartE2EDuration="2.676823227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:19.675270824 +0000 UTC m=+2741.222961321" watchObservedRunningTime="2026-10-30 01:28:19.676823227 +0000 UTC m=+2741.224513724"
Oct 30 01:28:20.064820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:20.064771    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"]
Oct 30 01:28:20.424530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:20.420728    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" podUID="439a5722-bfae-47e0-b389-c682c528531a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:20.424921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:20.424871    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" podUID="69499e2a-f2b6-4d79-8132-ed3b23a62752" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:20.584080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:20.584043    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" path="/var/lib/kubelet/pods/70785039-29f0-4fee-ae6e-3719b53420c0/volumes"
Oct 30 01:28:20.584918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:20.584898    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" path="/var/lib/kubelet/pods/ba1dc8cc-7b83-457f-a38a-f95d44e85dd4/volumes"
Oct 30 01:28:20.652090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:20.652051    5104 generic.go:356] "Generic (PLEG): container finished" podID="8c5545cc-2265-474c-978f-cb70a1efa5aa" containerID="d1e2db4f7aa14f14a737031b2067258c17936d7bedbf91fc58e667ce720ab961" exitCode=0
Oct 30 01:28:20.652544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:20.652134    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl" event={"ID":"8c5545cc-2265-474c-978f-cb70a1efa5aa","Type":"ContainerDied","Data":"d1e2db4f7aa14f14a737031b2067258c17936d7bedbf91fc58e667ce720ab961"}
Oct 30 01:28:21.131026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.130963    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"]
Oct 30 01:28:21.443177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.443067    5104 patch_prober.go:29] interesting pod/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:28:21.443177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.443145    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:28:21.665128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.665096    5104 generic.go:356] "Generic (PLEG): container finished" podID="bd41d294-abf2-4b68-bec1-f29971fca466" containerID="9d1ba47679fd8dfe61396c0aeded0563b7db0d82ba3db2e437c6017fc85ce242" exitCode=0
Oct 30 01:28:21.665663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.665169    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" event={"ID":"bd41d294-abf2-4b68-bec1-f29971fca466","Type":"ContainerDied","Data":"9d1ba47679fd8dfe61396c0aeded0563b7db0d82ba3db2e437c6017fc85ce242"}
Oct 30 01:28:21.665663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.665247    5104 scope.go:122] "RemoveContainer" containerID="b2002898df8c659365fc2e8744c05f4c17b9065db867f550e7a4a3826d6a7544"
Oct 30 01:28:21.892637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.891733    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk" podUID="60c35be3-ad83-400a-9c15-07f474664c3c" containerName="payload-processing" containerID="cri-o://4cbd8c349168d3b9db750c1cffb7009824907c32cfa865decc67a242c73a8848" gracePeriod=25
Oct 30 01:28:21.943048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:21.931205    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt" podUID="f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" containerName="payload-pre-processing" containerID="cri-o://ae8945706034fa659b09af0482fea480534c276d44f5cf7f8e5c3197623f7b75" gracePeriod=25
Oct 30 01:28:22.096482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.034528    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"]
Oct 30 01:28:22.096482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.036118    5104 cpu_manager.go:414] "containerMap: removing container" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy"
Oct 30 01:28:22.096482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.036157    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy"
Oct 30 01:28:22.096482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.036227    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy"
Oct 30 01:28:22.096482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.036241    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy"
Oct 30 01:28:22.096482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.036517    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1dc8cc-7b83-457f-a38a-f95d44e85dd4" containerName="istio-proxy"
Oct 30 01:28:22.096482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.036541    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy"
Oct 30 01:28:22.102762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.101786    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.131604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.122277    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"]
Oct 30 01:28:22.132963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.132173    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-679bd2-openshift-default-dockercfg-74xh8\""
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.273771    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-token\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.273892    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-podinfo\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.274086    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-credential-socket\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.274136    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-certs\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.274243    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-envoy\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.274306    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbscn\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-kube-api-access-nbscn\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.274342    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-socket\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.274409    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istiod-ca-cert\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.321671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.274447    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-data\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.466809    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-credential-socket\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.466883    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-certs\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.467004    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-envoy\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.467073    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbscn\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-kube-api-access-nbscn\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.467107    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-socket\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.467182    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istiod-ca-cert\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.467216    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-data\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.467303    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-token\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.479498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.467410    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-podinfo\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.520353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.520291    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-credential-socket\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.547156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.527914    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-socket\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.547440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.528161    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-certs\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.548471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.548429    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istiod-ca-cert\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.548951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.548928    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-data\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.595531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.590808    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-podinfo\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.666489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.659576    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-envoy\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:22.795482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.788033    5104 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 01:28:22.867017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.840250    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:28:22.894648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.877203    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:28:22.941953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:22.940324    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:28:23.160781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.153298    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbscn\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-kube-api-access-nbscn\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:23.241481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.230566    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-token\") pod \"e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") " pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:23.273930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.273879    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"]
Oct 30 01:28:23.294439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.294409    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-18a189-ghcxt"]
Oct 30 01:28:23.298902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.298875    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:23.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.343298    5104 generic.go:356] "Generic (PLEG): container finished" podID="60c35be3-ad83-400a-9c15-07f474664c3c" containerID="4cbd8c349168d3b9db750c1cffb7009824907c32cfa865decc67a242c73a8848" exitCode=0
Oct 30 01:28:23.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.343497    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk" event={"ID":"60c35be3-ad83-400a-9c15-07f474664c3c","Type":"ContainerDied","Data":"4cbd8c349168d3b9db750c1cffb7009824907c32cfa865decc67a242c73a8848"}
Oct 30 01:28:23.372032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.364554    5104 generic.go:356] "Generic (PLEG): container finished" podID="f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" containerID="ae8945706034fa659b09af0482fea480534c276d44f5cf7f8e5c3197623f7b75" exitCode=0
Oct 30 01:28:23.372032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.364854    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt" event={"ID":"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d","Type":"ContainerDied","Data":"ae8945706034fa659b09af0482fea480534c276d44f5cf7f8e5c3197623f7b75"}
Oct 30 01:28:23.592898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.592869    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:23.595372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.595284    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:28:23.656718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.656692    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:28:23.666686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.666652    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:28:23.685303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685257    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-envoy\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.685303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685311    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-certs\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685379    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-xks-ca\") pod \"8c5545cc-2265-474c-978f-cb70a1efa5aa\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685419    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-socket\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685548    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c35be3-ad83-400a-9c15-07f474664c3c-kube-api-access-zfp82\" (UniqueName: \"kubernetes.io/projected/60c35be3-ad83-400a-9c15-07f474664c3c-kube-api-access-zfp82\") pod \"60c35be3-ad83-400a-9c15-07f474664c3c\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685580    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c35be3-ad83-400a-9c15-07f474664c3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c35be3-ad83-400a-9c15-07f474664c3c-plugins-config-volume\") pod \"60c35be3-ad83-400a-9c15-07f474664c3c\" (UID: \"60c35be3-ad83-400a-9c15-07f474664c3c\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685635    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-data\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685707    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-credential-socket\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685734    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-istio-token\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685832    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5545cc-2265-474c-978f-cb70a1efa5aa-kube-api-access-k92n5\" (UniqueName: \"kubernetes.io/projected/8c5545cc-2265-474c-978f-cb70a1efa5aa-kube-api-access-k92n5\") pod \"8c5545cc-2265-474c-978f-cb70a1efa5aa\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.685974    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-openshift-ca\") pod \"8c5545cc-2265-474c-978f-cb70a1efa5aa\" (UID: \"8c5545cc-2265-474c-978f-cb70a1efa5aa\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.686039    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd41d294-abf2-4b68-bec1-f29971fca466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d294-abf2-4b68-bec1-f29971fca466-istiod-ca-cert\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.686077    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd41d294-abf2-4b68-bec1-f29971fca466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d294-abf2-4b68-bec1-f29971fca466-istio-podinfo\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.686157    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-plugins-config-volume\") pod \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") "
Oct 30 01:28:23.686249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.686212    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-kube-api-access-hnlj7\" (UniqueName: \"kubernetes.io/projected/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-kube-api-access-hnlj7\") pod \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\" (UID: \"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d\") "
Oct 30 01:28:23.687059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.686283    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-kube-api-access-xzwzv\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-kube-api-access-xzwzv\") pod \"bd41d294-abf2-4b68-bec1-f29971fca466\" (UID: \"bd41d294-abf2-4b68-bec1-f29971fca466\") "
Oct 30 01:28:23.688594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.687944    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-credential-socket" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:23.688742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.688715    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c35be3-ad83-400a-9c15-07f474664c3c-plugins-config-volume" pod "60c35be3-ad83-400a-9c15-07f474664c3c" (UID: "60c35be3-ad83-400a-9c15-07f474664c3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:23.689634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.689020    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-socket" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:23.689634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.689048    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-plugins-config-volume" pod "f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" (UID: "f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:23.689634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.689363    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-certs" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:23.690259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.690236    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd41d294-abf2-4b68-bec1-f29971fca466-istiod-ca-cert" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:23.692829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.692343    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-xks-ca" pod "8c5545cc-2265-474c-978f-cb70a1efa5aa" (UID: "8c5545cc-2265-474c-978f-cb70a1efa5aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:23.692829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.692579    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-data" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:23.702568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.702500    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-openshift-ca" pod "8c5545cc-2265-474c-978f-cb70a1efa5aa" (UID: "8c5545cc-2265-474c-978f-cb70a1efa5aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:23.705125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.705080    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd41d294-abf2-4b68-bec1-f29971fca466-istio-podinfo" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:23.705448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.705419    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-kube-api-access-hnlj7" pod "f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" (UID: "f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d"). InnerVolumeSpecName "kube-api-access-hnlj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:23.705775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.705736    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-kube-api-access-xzwzv" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "kube-api-access-xzwzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:23.707422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.707398    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c35be3-ad83-400a-9c15-07f474664c3c-kube-api-access-zfp82" pod "60c35be3-ad83-400a-9c15-07f474664c3c" (UID: "60c35be3-ad83-400a-9c15-07f474664c3c"). InnerVolumeSpecName "kube-api-access-zfp82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:23.709261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.709239    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-istio-token" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:23.712909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.712869    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5545cc-2265-474c-978f-cb70a1efa5aa-kube-api-access-k92n5" pod "8c5545cc-2265-474c-978f-cb70a1efa5aa" (UID: "8c5545cc-2265-474c-978f-cb70a1efa5aa"). InnerVolumeSpecName "kube-api-access-k92n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:23.713263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.713223    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-envoy" pod "bd41d294-abf2-4b68-bec1-f29971fca466" (UID: "bd41d294-abf2-4b68-bec1-f29971fca466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:23.789236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.789200    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.789485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.789444    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.789630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.789611    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.789722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.789707    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfp82\" (UniqueName: \"kubernetes.io/projected/60c35be3-ad83-400a-9c15-07f474664c3c-kube-api-access-zfp82\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.789815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.789798    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c35be3-ad83-400a-9c15-07f474664c3c-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.789907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.789891    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.789999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.789982    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790071    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790157    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92n5\" (UniqueName: \"kubernetes.io/projected/8c5545cc-2265-474c-978f-cb70a1efa5aa-kube-api-access-k92n5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790266 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790251    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5545cc-2265-474c-978f-cb70a1efa5aa-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790334    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd41d294-abf2-4b68-bec1-f29971fca466-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790426    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd41d294-abf2-4b68-bec1-f29971fca466-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790564    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790668    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlj7\" (UniqueName: \"kubernetes.io/projected/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d-kube-api-access-hnlj7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790754    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwzv\" (UniqueName: \"kubernetes.io/projected/bd41d294-abf2-4b68-bec1-f29971fca466-kube-api-access-xzwzv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.790843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.790833    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd41d294-abf2-4b68-bec1-f29971fca466-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:23.817576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.817545    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"]
Oct 30 01:28:23.829088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.829037    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:28:23.829208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.829147    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:28:23.829208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:23.829195    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:28:24.377794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.377628    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl" event={"ID":"8c5545cc-2265-474c-978f-cb70a1efa5aa","Type":"ContainerDied","Data":"e52541217bca0f1802a55143dbaefacfb1653cfea6eace49a3fdb379cbc41fcd"}
Oct 30 01:28:24.378314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.378053    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e52541217bca0f1802a55143dbaefacfb1653cfea6eace49a3fdb379cbc41fcd"
Oct 30 01:28:24.378314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.378186    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"
Oct 30 01:28:24.380121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.380044    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt" event={"ID":"f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d","Type":"ContainerDied","Data":"78810ba3457dcaffcf79a6258268ca1221cb53b417c59bcad2db9cea18cfdb12"}
Oct 30 01:28:24.380121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.380120    5104 scope.go:122] "RemoveContainer" containerID="ae8945706034fa659b09af0482fea480534c276d44f5cf7f8e5c3197623f7b75"
Oct 30 01:28:24.380324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.380284    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"
Oct 30 01:28:24.388303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.388263    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" event={"ID":"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3","Type":"ContainerStarted","Data":"6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511"}
Oct 30 01:28:24.388431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.388329    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" event={"ID":"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3","Type":"ContainerStarted","Data":"47e1b72781929e3fa6f2d6f6175362bf892a74a45a588fa8dba34c8ea7a8a652"}
Oct 30 01:28:24.391835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.391779    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk" event={"ID":"60c35be3-ad83-400a-9c15-07f474664c3c","Type":"ContainerDied","Data":"892c1443612a96a1982a18a93f1e4ad86039bae617b6ca4f250628b1a5449680"}
Oct 30 01:28:24.392129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.391905    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"
Oct 30 01:28:24.394491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.393910    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj" event={"ID":"bd41d294-abf2-4b68-bec1-f29971fca466","Type":"ContainerDied","Data":"dc71a96b78ff292046803b9047e0b7968b8d8be10a8b661d89c198c66074a161"}
Oct 30 01:28:24.394491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.394017    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"
Oct 30 01:28:24.407166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.406706    5104 scope.go:122] "RemoveContainer" containerID="4cbd8c349168d3b9db750c1cffb7009824907c32cfa865decc67a242c73a8848"
Oct 30 01:28:24.428560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.428537    5104 scope.go:122] "RemoveContainer" containerID="9d1ba47679fd8dfe61396c0aeded0563b7db0d82ba3db2e437c6017fc85ce242"
Oct 30 01:28:24.462854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.462796    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podStartSLOduration=3.462782756 podStartE2EDuration="3.462782756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:24.461915253 +0000 UTC m=+2746.009605761" watchObservedRunningTime="2026-10-30 01:28:24.462782756 +0000 UTC m=+2746.010473301"
Oct 30 01:28:24.501740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.501704    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"]
Oct 30 01:28:24.507444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.507412    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e2d6d3-777444d8ff-mrztt"]
Oct 30 01:28:24.536935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.536898    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"]
Oct 30 01:28:24.541934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.541904    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3ed60-openshift-default-7888cd9c47-gj2mj"]
Oct 30 01:28:24.733531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.732712    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b74a284-5196-49b9-b39a-ea75fe641f8d" path="/var/lib/kubelet/pods/6b74a284-5196-49b9-b39a-ea75fe641f8d/volumes"
Oct 30 01:28:24.733531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.733520    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" path="/var/lib/kubelet/pods/bd41d294-abf2-4b68-bec1-f29971fca466/volumes"
Oct 30 01:28:24.734139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.734118    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" path="/var/lib/kubelet/pods/f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d/volumes"
Oct 30 01:28:24.735024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.735004    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"]
Oct 30 01:28:24.735097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:24.735050    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e2d6d3-5bdbcf5d6d-9z2kk"]
Oct 30 01:28:25.065749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:25.065656    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" podUID="1bac07a8-d189-4a85-8961-7414d74492ce" containerName="payload-processing" containerID="cri-o://a0b2c60915418659922bd9dbf7d65f112d494ea003b010bd2e0f47c7c5f8224c" gracePeriod=25
Oct 30 01:28:25.302516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:25.302171    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:25.307375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:25.307342    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:25.411795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:25.411705    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:25.412876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:25.412850    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:28:26.143479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.137751    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s" podUID="eb0b22ab-d662-4495-ac26-f5ad3f84723c" containerName="payload-pre-processing" containerID="cri-o://112a1b69b44f6e6f3895df7fac66085399411bf5311452881b378817f87cbbeb" gracePeriod=25
Oct 30 01:28:26.432419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.432382    5104 generic.go:356] "Generic (PLEG): container finished" podID="eb0b22ab-d662-4495-ac26-f5ad3f84723c" containerID="112a1b69b44f6e6f3895df7fac66085399411bf5311452881b378817f87cbbeb" exitCode=0
Oct 30 01:28:26.432969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.432657    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s" event={"ID":"eb0b22ab-d662-4495-ac26-f5ad3f84723c","Type":"ContainerDied","Data":"112a1b69b44f6e6f3895df7fac66085399411bf5311452881b378817f87cbbeb"}
Oct 30 01:28:26.501991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.501962    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:28:26.551333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.551295    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0b22ab-d662-4495-ac26-f5ad3f84723c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0b22ab-d662-4495-ac26-f5ad3f84723c-plugins-config-volume\") pod \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") "
Oct 30 01:28:26.551745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.551719    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0b22ab-d662-4495-ac26-f5ad3f84723c-plugins-config-volume" pod "eb0b22ab-d662-4495-ac26-f5ad3f84723c" (UID: "eb0b22ab-d662-4495-ac26-f5ad3f84723c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:26.551898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.551750    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0b22ab-d662-4495-ac26-f5ad3f84723c-kube-api-access-kpg94\" (UniqueName: \"kubernetes.io/projected/eb0b22ab-d662-4495-ac26-f5ad3f84723c-kube-api-access-kpg94\") pod \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\" (UID: \"eb0b22ab-d662-4495-ac26-f5ad3f84723c\") "
Oct 30 01:28:26.552667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.552637    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0b22ab-d662-4495-ac26-f5ad3f84723c-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:26.556704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.556671    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0b22ab-d662-4495-ac26-f5ad3f84723c-kube-api-access-kpg94" pod "eb0b22ab-d662-4495-ac26-f5ad3f84723c" (UID: "eb0b22ab-d662-4495-ac26-f5ad3f84723c"). InnerVolumeSpecName "kube-api-access-kpg94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:26.586823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.586779    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c35be3-ad83-400a-9c15-07f474664c3c" path="/var/lib/kubelet/pods/60c35be3-ad83-400a-9c15-07f474664c3c/volumes"
Oct 30 01:28:26.654985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:26.654946    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpg94\" (UniqueName: \"kubernetes.io/projected/eb0b22ab-d662-4495-ac26-f5ad3f84723c-kube-api-access-kpg94\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:27.444410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:27.444374    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"
Oct 30 01:28:27.444410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:27.444391    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s" event={"ID":"eb0b22ab-d662-4495-ac26-f5ad3f84723c","Type":"ContainerDied","Data":"e8bdcdf06f141a1881ca2892711bf6c4db5e60c22cb56101f671fd4453182a6e"}
Oct 30 01:28:27.444978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:27.444489    5104 scope.go:122] "RemoveContainer" containerID="112a1b69b44f6e6f3895df7fac66085399411bf5311452881b378817f87cbbeb"
Oct 30 01:28:27.499389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:27.499350    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"]
Oct 30 01:28:27.510331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:27.510295    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-65846f6d8c-rht6s"]
Oct 30 01:28:28.038776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:28.038737    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"]
Oct 30 01:28:28.077440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:28.077395    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"]
Oct 30 01:28:28.582301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:28.582262    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0b22ab-d662-4495-ac26-f5ad3f84723c" path="/var/lib/kubelet/pods/eb0b22ab-d662-4495-ac26-f5ad3f84723c/volumes"
Oct 30 01:28:30.135439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.135399    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"]
Oct 30 01:28:30.136335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136319    5104 cpu_manager.go:414] "containerMap: removing container" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy"
Oct 30 01:28:30.136387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136337    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy"
Oct 30 01:28:30.136387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136347    5104 cpu_manager.go:414] "containerMap: removing container" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy"
Oct 30 01:28:30.136387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136352    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy"
Oct 30 01:28:30.136387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136366    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8c5545cc-2265-474c-978f-cb70a1efa5aa" containerName="revoke-keys"
Oct 30 01:28:30.136387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136372    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5545cc-2265-474c-978f-cb70a1efa5aa" containerName="revoke-keys"
Oct 30 01:28:30.136387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136387    5104 cpu_manager.go:414] "containerMap: removing container" podUID="60c35be3-ad83-400a-9c15-07f474664c3c" containerName="payload-processing"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136392    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c35be3-ad83-400a-9c15-07f474664c3c" containerName="payload-processing"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136403    5104 cpu_manager.go:414] "containerMap: removing container" podUID="eb0b22ab-d662-4495-ac26-f5ad3f84723c" containerName="payload-pre-processing"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136409    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0b22ab-d662-4495-ac26-f5ad3f84723c" containerName="payload-pre-processing"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136419    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" containerName="payload-pre-processing"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136424    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" containerName="payload-pre-processing"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136452    5104 cpu_manager.go:414] "containerMap: removing container" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136473    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136616    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136624    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd41d294-abf2-4b68-bec1-f29971fca466" containerName="istio-proxy"
Oct 30 01:28:30.136635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136634    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5545cc-2265-474c-978f-cb70a1efa5aa" containerName="revoke-keys"
Oct 30 01:28:30.136950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136642    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c35be3-ad83-400a-9c15-07f474664c3c" containerName="payload-processing"
Oct 30 01:28:30.136950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136650    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0b22ab-d662-4495-ac26-f5ad3f84723c" containerName="payload-pre-processing"
Oct 30 01:28:30.136950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136657    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3762b59-c5f7-4fb4-a9bc-2aa8a4c3016d" containerName="payload-pre-processing"
Oct 30 01:28:30.136950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.136662    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="70785039-29f0-4fee-ae6e-3719b53420c0" containerName="istio-proxy"
Oct 30 01:28:30.142415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.142393    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:30.159688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.159650    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"]
Oct 30 01:28:30.164839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.164808    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-679bd2\""
Oct 30 01:28:30.165050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.164854    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-679bd2-dockercfg-jz6tb\""
Oct 30 01:28:30.498550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.490032    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42544934-4a64-4f8b-8c7d-5b13dacbe54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:30.498550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.490144    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfp2k\" (UniqueName: \"kubernetes.io/projected/42544934-4a64-4f8b-8c7d-5b13dacbe54a-kube-api-access-mfp2k\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:30.498550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.490672    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" podUID="439a5722-bfae-47e0-b389-c682c528531a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:30.511721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.511613    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" podUID="69499e2a-f2b6-4d79-8132-ed3b23a62752" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:30.556588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.556527    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"]
Oct 30 01:28:30.647664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:30.627067    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.456857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.456819    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e39f16-1cf7-4311-9d97-4cd699f8d308-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.458195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.458161    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42544934-4a64-4f8b-8c7d-5b13dacbe54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:31.459570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.459550    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnhd\" (UniqueName: \"kubernetes.io/projected/e5e39f16-1cf7-4311-9d97-4cd699f8d308-kube-api-access-mvnhd\") pod \"payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.459935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.459915    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfp2k\" (UniqueName: \"kubernetes.io/projected/42544934-4a64-4f8b-8c7d-5b13dacbe54a-kube-api-access-mfp2k\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:31.536549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.502208    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:28:31.536549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.503032    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:28:31.548938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.548743    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"]
Oct 30 01:28:31.560430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.560393    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42544934-4a64-4f8b-8c7d-5b13dacbe54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:31.564848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.564798    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfp2k\" (UniqueName: \"kubernetes.io/projected/42544934-4a64-4f8b-8c7d-5b13dacbe54a-kube-api-access-mfp2k\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:31.568050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.568013    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"]
Oct 30 01:28:31.568293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.568261    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy" containerID="cri-o://aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427" gracePeriod=30
Oct 30 01:28:31.575327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.575291    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"]
Oct 30 01:28:31.575754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.575724    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy" containerID="cri-o://cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f" gracePeriod=30
Oct 30 01:28:31.581467 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.581420    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"]
Oct 30 01:28:31.586814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.586793    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.587256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.587235    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"]
Oct 30 01:28:31.600315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.600282    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:31.600434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.600374    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:28:31.646251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.646210    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" podUID="1bac07a8-d189-4a85-8961-7414d74492ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:31.655902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.655859    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e39f16-1cf7-4311-9d97-4cd699f8d308-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.656114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.656051    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnhd\" (UniqueName: \"kubernetes.io/projected/e5e39f16-1cf7-4311-9d97-4cd699f8d308-kube-api-access-mvnhd\") pod \"payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.656185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.656149    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.656273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.656256    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqkl\" (UniqueName: \"kubernetes.io/projected/465bd99a-d8e3-4a68-a705-28f568b7b099-kube-api-access-tqqkl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.656728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.656686    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.658282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.658248    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e39f16-1cf7-4311-9d97-4cd699f8d308-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.658899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.658872    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:31.678513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.678443    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnhd\" (UniqueName: \"kubernetes.io/projected/e5e39f16-1cf7-4311-9d97-4cd699f8d308-kube-api-access-mvnhd\") pod \"payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.761820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.760248    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.761820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.760314    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqkl\" (UniqueName: \"kubernetes.io/projected/465bd99a-d8e3-4a68-a705-28f568b7b099-kube-api-access-tqqkl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.761820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.760571    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.761820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.761547    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.764767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.764738    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.781263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.780824    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqkl\" (UniqueName: \"kubernetes.io/projected/465bd99a-d8e3-4a68-a705-28f568b7b099-kube-api-access-tqqkl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:31.864372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.864333    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:31.904656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:31.904623    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:32.002480 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:32.001631    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42544934_4a64_4f8b_8c7d_5b13dacbe54a.slice/crio-3a69fbfc0dd822349e273444acce2418a1c6f5e81cf36402b4614b4bc81c58ae WatchSource:0}: Error finding container 3a69fbfc0dd822349e273444acce2418a1c6f5e81cf36402b4614b4bc81c58ae: Status 404 returned error can't find the container with id 3a69fbfc0dd822349e273444acce2418a1c6f5e81cf36402b4614b4bc81c58ae
Oct 30 01:28:32.008687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.008647    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"]
Oct 30 01:28:32.184573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.184532    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"]
Oct 30 01:28:32.189637 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:32.189570    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e39f16_1cf7_4311_9d97_4cd699f8d308.slice/crio-5164de178b7a5f4e1a9678964e38a2dbe3ed9e1df390c61e176b1a06777d93d4 WatchSource:0}: Error finding container 5164de178b7a5f4e1a9678964e38a2dbe3ed9e1df390c61e176b1a06777d93d4: Status 404 returned error can't find the container with id 5164de178b7a5f4e1a9678964e38a2dbe3ed9e1df390c61e176b1a06777d93d4
Oct 30 01:28:32.364530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.364253    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"]
Oct 30 01:28:32.372619 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:32.372568    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465bd99a_d8e3_4a68_a705_28f568b7b099.slice/crio-1ae13af254a3d64e0cc6020bcaab98e2995a02f23d766d44efbbd01148e19b32 WatchSource:0}: Error finding container 1ae13af254a3d64e0cc6020bcaab98e2995a02f23d766d44efbbd01148e19b32: Status 404 returned error can't find the container with id 1ae13af254a3d64e0cc6020bcaab98e2995a02f23d766d44efbbd01148e19b32
Oct 30 01:28:32.467345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.467299    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 01:28:32.600580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.600546    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69499e2a-f2b6-4d79-8132-ed3b23a62752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69499e2a-f2b6-4d79-8132-ed3b23a62752-plugins-config-volume\") pod \"69499e2a-f2b6-4d79-8132-ed3b23a62752\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") "
Oct 30 01:28:32.600792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.600604    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69499e2a-f2b6-4d79-8132-ed3b23a62752-kube-api-access-hq8c7\" (UniqueName: \"kubernetes.io/projected/69499e2a-f2b6-4d79-8132-ed3b23a62752-kube-api-access-hq8c7\") pod \"69499e2a-f2b6-4d79-8132-ed3b23a62752\" (UID: \"69499e2a-f2b6-4d79-8132-ed3b23a62752\") "
Oct 30 01:28:32.601088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.601063    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69499e2a-f2b6-4d79-8132-ed3b23a62752-plugins-config-volume" pod "69499e2a-f2b6-4d79-8132-ed3b23a62752" (UID: "69499e2a-f2b6-4d79-8132-ed3b23a62752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:32.601713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.601689    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69499e2a-f2b6-4d79-8132-ed3b23a62752-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:32.603431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.603408    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69499e2a-f2b6-4d79-8132-ed3b23a62752-kube-api-access-hq8c7" pod "69499e2a-f2b6-4d79-8132-ed3b23a62752" (UID: "69499e2a-f2b6-4d79-8132-ed3b23a62752"). InnerVolumeSpecName "kube-api-access-hq8c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:32.637582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.637551    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 01:28:32.703603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.703564    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439a5722-bfae-47e0-b389-c682c528531a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a5722-bfae-47e0-b389-c682c528531a-plugins-config-volume\") pod \"439a5722-bfae-47e0-b389-c682c528531a\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") "
Oct 30 01:28:32.703786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.703769    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439a5722-bfae-47e0-b389-c682c528531a-kube-api-access-n8857\" (UniqueName: \"kubernetes.io/projected/439a5722-bfae-47e0-b389-c682c528531a-kube-api-access-n8857\") pod \"439a5722-bfae-47e0-b389-c682c528531a\" (UID: \"439a5722-bfae-47e0-b389-c682c528531a\") "
Oct 30 01:28:32.704080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.704052    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439a5722-bfae-47e0-b389-c682c528531a-plugins-config-volume" pod "439a5722-bfae-47e0-b389-c682c528531a" (UID: "439a5722-bfae-47e0-b389-c682c528531a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:32.704754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.704728    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439a5722-bfae-47e0-b389-c682c528531a-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:32.704933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.704916    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8c7\" (UniqueName: \"kubernetes.io/projected/69499e2a-f2b6-4d79-8132-ed3b23a62752-kube-api-access-hq8c7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:32.710275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.710141    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439a5722-bfae-47e0-b389-c682c528531a-kube-api-access-n8857" pod "439a5722-bfae-47e0-b389-c682c528531a" (UID: "439a5722-bfae-47e0-b389-c682c528531a"). InnerVolumeSpecName "kube-api-access-n8857". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:32.807704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:32.807662    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8857\" (UniqueName: \"kubernetes.io/projected/439a5722-bfae-47e0-b389-c682c528531a-kube-api-access-n8857\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:33.027436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.027398    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p" event={"ID":"e5e39f16-1cf7-4311-9d97-4cd699f8d308","Type":"ContainerStarted","Data":"e2041b7e740e7d8dcc800e586c653c354ba6f2c4d8b4d48b0893b350d4d8891d"}
Oct 30 01:28:33.027672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.027450    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p" event={"ID":"e5e39f16-1cf7-4311-9d97-4cd699f8d308","Type":"ContainerStarted","Data":"5164de178b7a5f4e1a9678964e38a2dbe3ed9e1df390c61e176b1a06777d93d4"}
Oct 30 01:28:33.027672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.027515    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:33.029555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.029522    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd" event={"ID":"42544934-4a64-4f8b-8c7d-5b13dacbe54a","Type":"ContainerStarted","Data":"c2c266ad76aed5b961d8403614c074870266ada76dc99c92594aeb55caca1396"}
Oct 30 01:28:33.029723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.029586    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:33.029723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.029603    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd" event={"ID":"42544934-4a64-4f8b-8c7d-5b13dacbe54a","Type":"ContainerStarted","Data":"3a69fbfc0dd822349e273444acce2418a1c6f5e81cf36402b4614b4bc81c58ae"}
Oct 30 01:28:33.031538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.031508    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7" event={"ID":"465bd99a-d8e3-4a68-a705-28f568b7b099","Type":"ContainerStarted","Data":"20948c049f7c19b4db8146b617a783ab428dbf0bec7aa3657edffc2742dcc4da"}
Oct 30 01:28:33.031666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.031564    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7" event={"ID":"465bd99a-d8e3-4a68-a705-28f568b7b099","Type":"ContainerStarted","Data":"1ae13af254a3d64e0cc6020bcaab98e2995a02f23d766d44efbbd01148e19b32"}
Oct 30 01:28:33.033008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.032985    5104 generic.go:356] "Generic (PLEG): container finished" podID="439a5722-bfae-47e0-b389-c682c528531a" containerID="6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6" exitCode=0
Oct 30 01:28:33.033119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.033017    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"
Oct 30 01:28:33.033119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.033068    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" event={"ID":"439a5722-bfae-47e0-b389-c682c528531a","Type":"ContainerDied","Data":"6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6"}
Oct 30 01:28:33.033237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.033116    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc" event={"ID":"439a5722-bfae-47e0-b389-c682c528531a","Type":"ContainerDied","Data":"1e97d04f40b76de5af77148e0bef0db0ad50d839479dc8d77e331e1b76b619b1"}
Oct 30 01:28:33.033237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.033134    5104 scope.go:122] "RemoveContainer" containerID="6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6"
Oct 30 01:28:33.034941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.034915    5104 generic.go:356] "Generic (PLEG): container finished" podID="69499e2a-f2b6-4d79-8132-ed3b23a62752" containerID="7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a" exitCode=0
Oct 30 01:28:33.035065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.034957    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"
Oct 30 01:28:33.035065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.034987    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" event={"ID":"69499e2a-f2b6-4d79-8132-ed3b23a62752","Type":"ContainerDied","Data":"7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a"}
Oct 30 01:28:33.035065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.035044    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp" event={"ID":"69499e2a-f2b6-4d79-8132-ed3b23a62752","Type":"ContainerDied","Data":"297c6a3179514b11993a9675b8ac1fe4132797a9e2e5fb43c1f5ea6bc4dd9970"}
Oct 30 01:28:33.039087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.039055    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" containerName="payload-processing" containerID="cri-o://b32ace3eb17a4329f02d287bf67e8f6ed033acbe7dbeb510664c5fe329560bf0" gracePeriod=25
Oct 30 01:28:33.050876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.050853    5104 scope.go:122] "RemoveContainer" containerID="6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6"
Oct 30 01:28:33.056893 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:33.056850    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6\": container with ID starting with 6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6 not found: ID does not exist" containerID="6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6"
Oct 30 01:28:33.057041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.056891    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6"} err="failed to get container status \"6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6\": rpc error: code = NotFound desc = could not find container \"6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6\": container with ID starting with 6d207f0583b86c857d1dfd513d73b397f6c0e2087ac771e07f7869fab82be2d6 not found: ID does not exist"
Oct 30 01:28:33.057041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.056917    5104 scope.go:122] "RemoveContainer" containerID="7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a"
Oct 30 01:28:33.068047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.067998    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p" podStartSLOduration=3.067983889 podStartE2EDuration="3.067983889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:33.063187766 +0000 UTC m=+2754.610878262" watchObservedRunningTime="2026-10-30 01:28:33.067983889 +0000 UTC m=+2754.615674385"
Oct 30 01:28:33.069821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.069799    5104 scope.go:122] "RemoveContainer" containerID="7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a"
Oct 30 01:28:33.070342 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:33.070168    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a\": container with ID starting with 7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a not found: ID does not exist" containerID="7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a"
Oct 30 01:28:33.070342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.070211    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a"} err="failed to get container status \"7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a\": rpc error: code = NotFound desc = could not find container \"7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a\": container with ID starting with 7309ee994bbfc2489ba9a2cb297c430cac88d4a8bfbf353fccb5bca154e02a0a not found: ID does not exist"
Oct 30 01:28:33.077831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.077807    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerName="payload-pre-processing" containerID="cri-o://d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7" gracePeriod=25
Oct 30 01:28:33.102227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.102171    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7" podStartSLOduration=2.10215406 podStartE2EDuration="2.10215406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:33.098945604 +0000 UTC m=+2754.646636101" watchObservedRunningTime="2026-10-30 01:28:33.10215406 +0000 UTC m=+2754.649844556"
Oct 30 01:28:33.141199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.141124    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"]
Oct 30 01:28:33.149444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.149407    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba3d7b-69dccb6f88-hjxhp"]
Oct 30 01:28:33.181483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.181261    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd" podStartSLOduration=3.181222076 podStartE2EDuration="3.181222076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:33.170109259 +0000 UTC m=+2754.717799756" watchObservedRunningTime="2026-10-30 01:28:33.181222076 +0000 UTC m=+2754.728912574"
Oct 30 01:28:33.266092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.266054    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"]
Oct 30 01:28:33.272374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:33.272337    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba3d7b-7666cb7f95-wgmzc"]
Oct 30 01:28:34.050714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:34.050670    5104 generic.go:356] "Generic (PLEG): container finished" podID="465bd99a-d8e3-4a68-a705-28f568b7b099" containerID="20948c049f7c19b4db8146b617a783ab428dbf0bec7aa3657edffc2742dcc4da" exitCode=0
Oct 30 01:28:34.051223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:34.050899    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7" event={"ID":"465bd99a-d8e3-4a68-a705-28f568b7b099","Type":"ContainerDied","Data":"20948c049f7c19b4db8146b617a783ab428dbf0bec7aa3657edffc2742dcc4da"}
Oct 30 01:28:34.581929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:34.581887    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439a5722-bfae-47e0-b389-c682c528531a" path="/var/lib/kubelet/pods/439a5722-bfae-47e0-b389-c682c528531a/volumes"
Oct 30 01:28:34.582618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:34.582591    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69499e2a-f2b6-4d79-8132-ed3b23a62752" path="/var/lib/kubelet/pods/69499e2a-f2b6-4d79-8132-ed3b23a62752/volumes"
Oct 30 01:28:35.277715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.277680    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:35.374256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.374159    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-xks-ca\") pod \"465bd99a-d8e3-4a68-a705-28f568b7b099\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") "
Oct 30 01:28:35.374439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.374294    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-openshift-ca\") pod \"465bd99a-d8e3-4a68-a705-28f568b7b099\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") "
Oct 30 01:28:35.374439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.374357    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465bd99a-d8e3-4a68-a705-28f568b7b099-kube-api-access-tqqkl\" (UniqueName: \"kubernetes.io/projected/465bd99a-d8e3-4a68-a705-28f568b7b099-kube-api-access-tqqkl\") pod \"465bd99a-d8e3-4a68-a705-28f568b7b099\" (UID: \"465bd99a-d8e3-4a68-a705-28f568b7b099\") "
Oct 30 01:28:35.374990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.374871    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-openshift-ca" pod "465bd99a-d8e3-4a68-a705-28f568b7b099" (UID: "465bd99a-d8e3-4a68-a705-28f568b7b099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:35.375376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.375348    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.377325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.377292    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-xks-ca" pod "465bd99a-d8e3-4a68-a705-28f568b7b099" (UID: "465bd99a-d8e3-4a68-a705-28f568b7b099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:35.377607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.377585    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465bd99a-d8e3-4a68-a705-28f568b7b099-kube-api-access-tqqkl" pod "465bd99a-d8e3-4a68-a705-28f568b7b099" (UID: "465bd99a-d8e3-4a68-a705-28f568b7b099"). InnerVolumeSpecName "kube-api-access-tqqkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:35.477117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.477077    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqkl\" (UniqueName: \"kubernetes.io/projected/465bd99a-d8e3-4a68-a705-28f568b7b099-kube-api-access-tqqkl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.477117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:35.477113    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465bd99a-d8e3-4a68-a705-28f568b7b099-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:36.076211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:36.076183    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"
Oct 30 01:28:36.076470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:36.076176    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7" event={"ID":"465bd99a-d8e3-4a68-a705-28f568b7b099","Type":"ContainerDied","Data":"1ae13af254a3d64e0cc6020bcaab98e2995a02f23d766d44efbbd01148e19b32"}
Oct 30 01:28:36.076470 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:36.076348    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae13af254a3d64e0cc6020bcaab98e2995a02f23d766d44efbbd01148e19b32"
Oct 30 01:28:36.938546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:36.938479    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:28:36.961535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:36.961504    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:28:37.115847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.115595    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-kube-api-access-cdbhz\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-kube-api-access-cdbhz\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.115847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.115673    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa93bf74-fd15-47c7-8472-401b203e726e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93bf74-fd15-47c7-8472-401b203e726e-istio-podinfo\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.115747    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-credential-socket\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116208    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-data\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116265    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-envoy\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116309    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47333922-85a0-4b2f-ba9e-f4766827a5a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47333922-85a0-4b2f-ba9e-f4766827a5a9-istiod-ca-cert\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116360    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-data\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116408    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-podinfo\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116593    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-kube-api-access-kx5wk\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-kube-api-access-kx5wk\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116675    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-istio-token\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116714    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa93bf74-fd15-47c7-8472-401b203e726e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93bf74-fd15-47c7-8472-401b203e726e-istiod-ca-cert\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116759    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-certs\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116831    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-envoy\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.116942    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-socket\") pod \"fa93bf74-fd15-47c7-8472-401b203e726e\" (UID: \"fa93bf74-fd15-47c7-8472-401b203e726e\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.117027    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-socket\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.117065    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-certs\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.117120    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-token\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-token\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.118503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.117181    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-credential-socket\") pod \"47333922-85a0-4b2f-ba9e-f4766827a5a9\" (UID: \"47333922-85a0-4b2f-ba9e-f4766827a5a9\") "
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.122556    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-data" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.123008    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa93bf74-fd15-47c7-8472-401b203e726e-istiod-ca-cert" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.123147    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-kube-api-access-kx5wk" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "kube-api-access-kx5wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.123534    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-socket" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.123738    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-socket" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.123916    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-data" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.124232    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-credential-socket" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.125804    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-certs" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.125908    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-kube-api-access-cdbhz" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "kube-api-access-cdbhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.127135    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-credential-socket" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.127323    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-certs" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.127518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.127487    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47333922-85a0-4b2f-ba9e-f4766827a5a9-istiod-ca-cert" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:37.145191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.145153    5104 generic.go:356] "Generic (PLEG): container finished" podID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerID="cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f" exitCode=0
Oct 30 01:28:37.145537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.145398    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" event={"ID":"47333922-85a0-4b2f-ba9e-f4766827a5a9","Type":"ContainerDied","Data":"cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f"}
Oct 30 01:28:37.146916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.146855    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc" event={"ID":"47333922-85a0-4b2f-ba9e-f4766827a5a9","Type":"ContainerDied","Data":"a6318593342b51f53b8288e6d86a3ab80056618d5e346fc376b60ced7e2204dc"}
Oct 30 01:28:37.146916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.146896    5104 scope.go:122] "RemoveContainer" containerID="cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f"
Oct 30 01:28:37.150423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.148553    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"
Oct 30 01:28:37.151347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.150428    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa93bf74-fd15-47c7-8472-401b203e726e-istio-podinfo" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:37.152548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.152331    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-podinfo" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:37.153005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.152982    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-envoy" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.153356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.153331    5104 generic.go:356] "Generic (PLEG): container finished" podID="fa93bf74-fd15-47c7-8472-401b203e726e" containerID="aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427" exitCode=0
Oct 30 01:28:37.153663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.153518    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" event={"ID":"fa93bf74-fd15-47c7-8472-401b203e726e","Type":"ContainerDied","Data":"aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427"}
Oct 30 01:28:37.153663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.153553    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"
Oct 30 01:28:37.153663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.153569    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s" event={"ID":"fa93bf74-fd15-47c7-8472-401b203e726e","Type":"ContainerDied","Data":"91b2954f06a9960144cdfdd64967b6aed8ae2ca072c48b5a52ac968bf3b2b64a"}
Oct 30 01:28:37.163572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.162818    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-envoy" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:37.167914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.167734    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-istio-token" pod "fa93bf74-fd15-47c7-8472-401b203e726e" (UID: "fa93bf74-fd15-47c7-8472-401b203e726e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:37.170107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.168975    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-token" pod "47333922-85a0-4b2f-ba9e-f4766827a5a9" (UID: "47333922-85a0-4b2f-ba9e-f4766827a5a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:37.179731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.173519    5104 scope.go:122] "RemoveContainer" containerID="cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f"
Oct 30 01:28:37.180921 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:37.180889    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f\": container with ID starting with cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f not found: ID does not exist" containerID="cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f"
Oct 30 01:28:37.181065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.180930    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f"} err="failed to get container status \"cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f\": rpc error: code = NotFound desc = could not find container \"cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f\": container with ID starting with cfb5f5a24e71ce3c021cbaa87b7ef306529b0e3a74ada9b032ee2a921f77658f not found: ID does not exist"
Oct 30 01:28:37.181065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.180953    5104 scope.go:122] "RemoveContainer" containerID="aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427"
Oct 30 01:28:37.200256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.199833    5104 scope.go:122] "RemoveContainer" containerID="9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371"
Oct 30 01:28:37.220796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.220722    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx5wk\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-kube-api-access-kx5wk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.220796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.220749    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93bf74-fd15-47c7-8472-401b203e726e-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.220796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.220760    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93bf74-fd15-47c7-8472-401b203e726e-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.220796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.220768    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221085    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221113    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221123    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221132    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221140    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221148    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221427    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdbhz\" (UniqueName: \"kubernetes.io/projected/47333922-85a0-4b2f-ba9e-f4766827a5a9-kube-api-access-cdbhz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221441    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93bf74-fd15-47c7-8472-401b203e726e-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221450    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221486    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221496    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221503    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47333922-85a0-4b2f-ba9e-f4766827a5a9-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221511    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93bf74-fd15-47c7-8472-401b203e726e-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.221520    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47333922-85a0-4b2f-ba9e-f4766827a5a9-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:37.236490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.236434    5104 scope.go:122] "RemoveContainer" containerID="aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427"
Oct 30 01:28:37.237011 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:37.236882    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427\": container with ID starting with aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427 not found: ID does not exist" containerID="aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427"
Oct 30 01:28:37.237011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.236925    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427"} err="failed to get container status \"aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427\": rpc error: code = NotFound desc = could not find container \"aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427\": container with ID starting with aa3c7420dff11751a6976978c687a04d0031fde5299c38055abb4dbd6aef6427 not found: ID does not exist"
Oct 30 01:28:37.237011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.236948    5104 scope.go:122] "RemoveContainer" containerID="9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371"
Oct 30 01:28:37.237321 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:37.237284    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371\": container with ID starting with 9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371 not found: ID does not exist" containerID="9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371"
Oct 30 01:28:37.237400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.237317    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371"} err="failed to get container status \"9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371\": rpc error: code = NotFound desc = could not find container \"9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371\": container with ID starting with 9f01b51a2e81c8786866405521303b067908f9a8866878ce2ee648dd8c560371 not found: ID does not exist"
Oct 30 01:28:37.500804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.500765    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"]
Oct 30 01:28:37.509093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.509059    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-j52fc"]
Oct 30 01:28:37.535700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.535655    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"]
Oct 30 01:28:37.545725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:37.545685    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-tbk6s"]
Oct 30 01:28:38.581616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:38.581580    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" path="/var/lib/kubelet/pods/47333922-85a0-4b2f-ba9e-f4766827a5a9/volumes"
Oct 30 01:28:38.582212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:38.582193    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" path="/var/lib/kubelet/pods/fa93bf74-fd15-47c7-8472-401b203e726e/volumes"
Oct 30 01:28:39.028047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:39.028011    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"]
Oct 30 01:28:39.039110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:39.039079    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a36e0284-cqt4n"]
Oct 30 01:28:39.107061 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:39.107019    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:28:40.583471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:40.583426    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5555030-b09d-4df4-9632-7575b8d2f19f" path="/var/lib/kubelet/pods/a5555030-b09d-4df4-9632-7575b8d2f19f/volumes"
Oct 30 01:28:41.223310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:41.223271    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:41.223525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:41.223327    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:41.645890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:41.645800    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" podUID="1bac07a8-d189-4a85-8961-7414d74492ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:41.795404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:41.795371    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"]
Oct 30 01:28:41.826237 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:41.826200    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"]
Oct 30 01:28:44.059152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:44.059121    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:28:44.059613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:44.059192    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:28:45.248982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.248948    5104 generic.go:356] "Generic (PLEG): container finished" podID="1bac07a8-d189-4a85-8961-7414d74492ce" containerID="a0b2c60915418659922bd9dbf7d65f112d494ea003b010bd2e0f47c7c5f8224c" exitCode=0
Oct 30 01:28:45.249385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.249028    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" event={"ID":"1bac07a8-d189-4a85-8961-7414d74492ce","Type":"ContainerDied","Data":"a0b2c60915418659922bd9dbf7d65f112d494ea003b010bd2e0f47c7c5f8224c"}
Oct 30 01:28:45.418370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.418336    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:28:45.459887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.459834    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bac07a8-d189-4a85-8961-7414d74492ce-kube-api-access-rt5s9\" (UniqueName: \"kubernetes.io/projected/1bac07a8-d189-4a85-8961-7414d74492ce-kube-api-access-rt5s9\") pod \"1bac07a8-d189-4a85-8961-7414d74492ce\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") "
Oct 30 01:28:45.460068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.460012    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bac07a8-d189-4a85-8961-7414d74492ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac07a8-d189-4a85-8961-7414d74492ce-plugins-config-volume\") pod \"1bac07a8-d189-4a85-8961-7414d74492ce\" (UID: \"1bac07a8-d189-4a85-8961-7414d74492ce\") "
Oct 30 01:28:45.461136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.461109    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bac07a8-d189-4a85-8961-7414d74492ce-plugins-config-volume" pod "1bac07a8-d189-4a85-8961-7414d74492ce" (UID: "1bac07a8-d189-4a85-8961-7414d74492ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:45.473512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.463742    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bac07a8-d189-4a85-8961-7414d74492ce-kube-api-access-rt5s9" pod "1bac07a8-d189-4a85-8961-7414d74492ce" (UID: "1bac07a8-d189-4a85-8961-7414d74492ce"). InnerVolumeSpecName "kube-api-access-rt5s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:45.562669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.562624    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt5s9\" (UniqueName: \"kubernetes.io/projected/1bac07a8-d189-4a85-8961-7414d74492ce-kube-api-access-rt5s9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:45.562669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:45.562662    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac07a8-d189-4a85-8961-7414d74492ce-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:46.260701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.260668    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"
Oct 30 01:28:46.261269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.260701    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68" event={"ID":"1bac07a8-d189-4a85-8961-7414d74492ce","Type":"ContainerDied","Data":"e2de0801ad5b57e6445ab7530870745ec9b7d42779e86b0c3ccd486701f1d930"}
Oct 30 01:28:46.261269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.260767    5104 scope.go:122] "RemoveContainer" containerID="a0b2c60915418659922bd9dbf7d65f112d494ea003b010bd2e0f47c7c5f8224c"
Oct 30 01:28:46.311835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.311791    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"]
Oct 30 01:28:46.321831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.321796    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-18a189-d4495f868-nsp68"]
Oct 30 01:28:46.581734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.581646    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bac07a8-d189-4a85-8961-7414d74492ce" path="/var/lib/kubelet/pods/1bac07a8-d189-4a85-8961-7414d74492ce/volumes"
Oct 30 01:28:46.795748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.795708    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerName="payload-processing" containerID="cri-o://18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82" gracePeriod=25
Oct 30 01:28:46.827170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:46.827123    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" podUID="4054d583-1459-402e-a08f-477bdd2e067a" containerName="payload-pre-processing" containerID="cri-o://2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b" gracePeriod=25
Oct 30 01:28:47.088676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:47.088641    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:50.029742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:50.029701    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"]
Oct 30 01:28:50.041932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:50.041892    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c09de66-jgfnr"]
Oct 30 01:28:50.635876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:50.633106    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a57814a-ee26-4472-862e-a0343b717010" path="/var/lib/kubelet/pods/9a57814a-ee26-4472-862e-a0343b717010/volumes"
Oct 30 01:28:50.663263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:50.663226    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"]
Oct 30 01:28:50.673622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:50.673587    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba3d7b-mck2l"]
Oct 30 01:28:51.192172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:51.192142    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"]
Oct 30 01:28:51.261739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:51.261644    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:51.264650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:51.264608    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:51.321486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:51.300145    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-262230-j9mpl"]
Oct 30 01:28:51.830386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:51.830346    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"]
Oct 30 01:28:51.841578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:51.841543    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e2d6d3-kxnnm"]
Oct 30 01:28:52.585000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:52.584957    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e8f361-3b6c-479b-ae99-e8769cae1c46" path="/var/lib/kubelet/pods/18e8f361-3b6c-479b-ae99-e8769cae1c46/volumes"
Oct 30 01:28:52.585906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:52.585880    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5545cc-2265-474c-978f-cb70a1efa5aa" path="/var/lib/kubelet/pods/8c5545cc-2265-474c-978f-cb70a1efa5aa/volumes"
Oct 30 01:28:52.586684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:52.586658    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d13cee-ed78-493e-88a4-464bc231f1de" path="/var/lib/kubelet/pods/99d13cee-ed78-493e-88a4-464bc231f1de/volumes"
Oct 30 01:28:53.464100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.462690    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:28:53.546480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.540564    5104 scope.go:122] "RemoveContainer" containerID="381db6d276acd3ba761d343e6ec7272f2817124e61ac7d2b96dfc3bbf083d852"
Oct 30 01:28:53.556498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.548381    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-plugins-config-volume\") pod \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") "
Oct 30 01:28:53.578482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.563594    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-plugins-config-volume" pod "94a90c34-26ab-4eec-bab8-2c1c6a26bb76" (UID: "94a90c34-26ab-4eec-bab8-2c1c6a26bb76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:53.578482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.563938    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-kube-api-access-qgq28\" (UniqueName: \"kubernetes.io/projected/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-kube-api-access-qgq28\") pod \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\" (UID: \"94a90c34-26ab-4eec-bab8-2c1c6a26bb76\") "
Oct 30 01:28:53.578482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.565024    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:53.647315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.647247    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-kube-api-access-qgq28" pod "94a90c34-26ab-4eec-bab8-2c1c6a26bb76" (UID: "94a90c34-26ab-4eec-bab8-2c1c6a26bb76"). InnerVolumeSpecName "kube-api-access-qgq28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.669452    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"]
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670031    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgq28\" (UniqueName: \"kubernetes.io/projected/94a90c34-26ab-4eec-bab8-2c1c6a26bb76-kube-api-access-qgq28\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670657    5104 cpu_manager.go:414] "containerMap: removing container" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670674    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670728    5104 cpu_manager.go:414] "containerMap: removing container" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670736    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670758    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1bac07a8-d189-4a85-8961-7414d74492ce" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670766    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bac07a8-d189-4a85-8961-7414d74492ce" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670784    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670791    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670812    5104 cpu_manager.go:414] "containerMap: removing container" podUID="439a5722-bfae-47e0-b389-c682c528531a" containerName="payload-pre-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670820    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439a5722-bfae-47e0-b389-c682c528531a" containerName="payload-pre-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670858    5104 cpu_manager.go:414] "containerMap: removing container" podUID="69499e2a-f2b6-4d79-8132-ed3b23a62752" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670864    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69499e2a-f2b6-4d79-8132-ed3b23a62752" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670893    5104 cpu_manager.go:414] "containerMap: removing container" podUID="465bd99a-d8e3-4a68-a705-28f568b7b099" containerName="revoke-keys"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.670900    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465bd99a-d8e3-4a68-a705-28f568b7b099" containerName="revoke-keys"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.671061    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="47333922-85a0-4b2f-ba9e-f4766827a5a9" containerName="istio-proxy"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.671072    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bac07a8-d189-4a85-8961-7414d74492ce" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.671081    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.671092    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="439a5722-bfae-47e0-b389-c682c528531a" containerName="payload-pre-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.671106    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="69499e2a-f2b6-4d79-8132-ed3b23a62752" containerName="payload-processing"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.671118    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="465bd99a-d8e3-4a68-a705-28f568b7b099" containerName="revoke-keys"
Oct 30 01:28:53.671342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.671130    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" containerName="payload-processing"
Oct 30 01:28:53.718722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.718694    5104 scope.go:122] "RemoveContainer" containerID="5f9120c2800b7fd6bd4de4edfd22780a74e95450657fb323a76c1f29fe2ad990"
Oct 30 01:28:53.719733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.719707    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"]
Oct 30 01:28:53.719876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.719758    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"]
Oct 30 01:28:53.720698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.720677    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy"
Oct 30 01:28:53.720698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.720698    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy"
Oct 30 01:28:53.720967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.720952    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa93bf74-fd15-47c7-8472-401b203e726e" containerName="istio-proxy"
Oct 30 01:28:53.722270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.722244    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:53.731833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.731802    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"]
Oct 30 01:28:53.732016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.731915    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:53.744154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.744118    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-29bb56\""
Oct 30 01:28:53.744316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.744299    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-29bb56-dockercfg-gscvc\""
Oct 30 01:28:53.749334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.749202    5104 scope.go:122] "RemoveContainer" containerID="5518980b575ee7cf153aeab75138fa4d3cb1545e201180cc3d0b6d45b67aee94"
Oct 30 01:28:53.771367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.771310    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbc9\" (UniqueName: \"kubernetes.io/projected/93218cf5-525b-4b93-8ef4-63c8ca95408d-kube-api-access-xnbc9\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:53.771555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.771526    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ddr\" (UniqueName: \"kubernetes.io/projected/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-kube-api-access-s6ddr\") pod \"payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:53.771754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.771738    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:53.771893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.771876    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93218cf5-525b-4b93-8ef4-63c8ca95408d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:53.773022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.773002    5104 scope.go:122] "RemoveContainer" containerID="b32ace3eb17a4329f02d287bf67e8f6ed033acbe7dbeb510664c5fe329560bf0"
Oct 30 01:28:53.785857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.785831    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:28:53.788777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.788753    5104 scope.go:122] "RemoveContainer" containerID="5356b7dd1e12fdd46191562ac6673fa73e56942f1134fb485051802cc4edf0f1"
Oct 30 01:28:53.801093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.801067    5104 scope.go:122] "RemoveContainer" containerID="a6aa2a2b46ae40f12062e4c01f0585e2c1acfce3ff60690951dff6f46bd2b8a4"
Oct 30 01:28:53.874358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.873380    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb63edbc-f6c8-4581-9537-f3b49f54165e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb63edbc-f6c8-4581-9537-f3b49f54165e-plugins-config-volume\") pod \"fb63edbc-f6c8-4581-9537-f3b49f54165e\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") "
Oct 30 01:28:53.874358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.873664    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb63edbc-f6c8-4581-9537-f3b49f54165e-kube-api-access-sv97t\" (UniqueName: \"kubernetes.io/projected/fb63edbc-f6c8-4581-9537-f3b49f54165e-kube-api-access-sv97t\") pod \"fb63edbc-f6c8-4581-9537-f3b49f54165e\" (UID: \"fb63edbc-f6c8-4581-9537-f3b49f54165e\") "
Oct 30 01:28:53.874621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.874478    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93218cf5-525b-4b93-8ef4-63c8ca95408d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:53.874621 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.874522    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbc9\" (UniqueName: \"kubernetes.io/projected/93218cf5-525b-4b93-8ef4-63c8ca95408d-kube-api-access-xnbc9\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:53.874935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.874653    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ddr\" (UniqueName: \"kubernetes.io/projected/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-kube-api-access-s6ddr\") pod \"payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:53.874935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.874766    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:53.875875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.875583    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:53.876005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.875965    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb63edbc-f6c8-4581-9537-f3b49f54165e-plugins-config-volume" pod "fb63edbc-f6c8-4581-9537-f3b49f54165e" (UID: "fb63edbc-f6c8-4581-9537-f3b49f54165e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:53.877708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.877676    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93218cf5-525b-4b93-8ef4-63c8ca95408d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:53.890786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.890753    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb63edbc-f6c8-4581-9537-f3b49f54165e-kube-api-access-sv97t" pod "fb63edbc-f6c8-4581-9537-f3b49f54165e" (UID: "fb63edbc-f6c8-4581-9537-f3b49f54165e"). InnerVolumeSpecName "kube-api-access-sv97t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:53.891029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.891011    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"
Oct 30 01:28:53.891268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.891078    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" event={"ID":"94a90c34-26ab-4eec-bab8-2c1c6a26bb76","Type":"ContainerDied","Data":"b32ace3eb17a4329f02d287bf67e8f6ed033acbe7dbeb510664c5fe329560bf0"}
Oct 30 01:28:53.896197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.895086    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf" event={"ID":"94a90c34-26ab-4eec-bab8-2c1c6a26bb76","Type":"ContainerDied","Data":"d5ecead2a6b5c4b540ead558869b86b644a57ef480af793cc31b2b33f75204a3"}
Oct 30 01:28:53.898467 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.898425    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ddr\" (UniqueName: \"kubernetes.io/projected/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-kube-api-access-s6ddr\") pod \"payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:53.900249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.899773    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbc9\" (UniqueName: \"kubernetes.io/projected/93218cf5-525b-4b93-8ef4-63c8ca95408d-kube-api-access-xnbc9\") pod \"payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:53.900775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.900749    5104 generic.go:356] "Generic (PLEG): container finished" podID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerID="d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7" exitCode=0
Oct 30 01:28:53.900942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.900893    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" event={"ID":"fb63edbc-f6c8-4581-9537-f3b49f54165e","Type":"ContainerDied","Data":"d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7"}
Oct 30 01:28:53.900942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.900926    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"
Oct 30 01:28:53.901039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.900957    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm" event={"ID":"fb63edbc-f6c8-4581-9537-f3b49f54165e","Type":"ContainerDied","Data":"ad560debe903913fc761952c9e39d3bbc9aa170d930879c166c10aba67422f17"}
Oct 30 01:28:53.901039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.900998    5104 scope.go:122] "RemoveContainer" containerID="d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7"
Oct 30 01:28:53.926904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.926853    5104 scope.go:122] "RemoveContainer" containerID="d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7"
Oct 30 01:28:53.927398 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:28:53.927344    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7\": container with ID starting with d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7 not found: ID does not exist" containerID="d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7"
Oct 30 01:28:53.927535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.927402    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7"} err="failed to get container status \"d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7\": rpc error: code = NotFound desc = could not find container \"d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7\": container with ID starting with d72ea2dee25cb5aff1b69509f60e5fe099e9ca7c9e33d638470babe21db086f7 not found: ID does not exist"
Oct 30 01:28:53.959733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.959704    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"]
Oct 30 01:28:53.964616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.964570    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-262230-9f68f6684-zfxzf"]
Oct 30 01:28:53.976791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.976762    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb63edbc-f6c8-4581-9537-f3b49f54165e-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:53.976791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.976791    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv97t\" (UniqueName: \"kubernetes.io/projected/fb63edbc-f6c8-4581-9537-f3b49f54165e-kube-api-access-sv97t\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:28:53.999268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:53.999235    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"]
Oct 30 01:28:54.004288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.004217    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-262230-6c79cfb8b7-z2wnm"]
Oct 30 01:28:54.066342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.066311    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:54.084792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.084765    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:54.584810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.584732    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a90c34-26ab-4eec-bab8-2c1c6a26bb76" path="/var/lib/kubelet/pods/94a90c34-26ab-4eec-bab8-2c1c6a26bb76/volumes"
Oct 30 01:28:54.585320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.585303    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" path="/var/lib/kubelet/pods/fb63edbc-f6c8-4581-9537-f3b49f54165e/volumes"
Oct 30 01:28:54.591993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.591954    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"]
Oct 30 01:28:54.593704 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:54.593674    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93218cf5_525b_4b93_8ef4_63c8ca95408d.slice/crio-93a7706c7f8afddd07a96feb59a992e63fb8fd6a29a2be97c10a32d12bbf94cf WatchSource:0}: Error finding container 93a7706c7f8afddd07a96feb59a992e63fb8fd6a29a2be97c10a32d12bbf94cf: Status 404 returned error can't find the container with id 93a7706c7f8afddd07a96feb59a992e63fb8fd6a29a2be97c10a32d12bbf94cf
Oct 30 01:28:54.612404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.612376    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"]
Oct 30 01:28:54.614614 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:28:54.614576    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1863d77a_baf2_4c78_bfe0_9b9b15bf319e.slice/crio-e25a47ccc48ef788a21b4bde5fd99150988db99e81a954e338407a856052583d WatchSource:0}: Error finding container e25a47ccc48ef788a21b4bde5fd99150988db99e81a954e338407a856052583d: Status 404 returned error can't find the container with id e25a47ccc48ef788a21b4bde5fd99150988db99e81a954e338407a856052583d
Oct 30 01:28:54.735474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.733846    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"]
Oct 30 01:28:54.735474 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.734182    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" containerID="cri-o://3e656146389bc57a604490506a0f474ddf3f560281ea067ecc593e52785cefdd" gracePeriod=30
Oct 30 01:28:54.739951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.739921    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"]
Oct 30 01:28:54.740140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.740120    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy" containerID="cri-o://cc1ef63d8cbf4ba76e887d328e825e3386a7e13a47d86a08d4b1e605c802ceb9" gracePeriod=30
Oct 30 01:28:54.815826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.815783    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" podUID="4054d583-1459-402e-a08f-477bdd2e067a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:54.926878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.926843    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b" event={"ID":"1863d77a-baf2-4c78-bfe0-9b9b15bf319e","Type":"ContainerStarted","Data":"470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78"}
Oct 30 01:28:54.927093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.926898    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b" event={"ID":"1863d77a-baf2-4c78-bfe0-9b9b15bf319e","Type":"ContainerStarted","Data":"e25a47ccc48ef788a21b4bde5fd99150988db99e81a954e338407a856052583d"}
Oct 30 01:28:54.927093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.926921    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:28:54.931025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.930996    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2" event={"ID":"93218cf5-525b-4b93-8ef4-63c8ca95408d","Type":"ContainerStarted","Data":"07e5c3a190c79a42a8203f601424914d8344f038d523b722809b6fed1be4c941"}
Oct 30 01:28:54.931168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.931045    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2" event={"ID":"93218cf5-525b-4b93-8ef4-63c8ca95408d","Type":"ContainerStarted","Data":"93a7706c7f8afddd07a96feb59a992e63fb8fd6a29a2be97c10a32d12bbf94cf"}
Oct 30 01:28:54.931168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.931099    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:28:54.965611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:54.965553    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b" podStartSLOduration=1.965537394 podStartE2EDuration="1.965537394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:54.963034411 +0000 UTC m=+2776.510724904" watchObservedRunningTime="2026-10-30 01:28:54.965537394 +0000 UTC m=+2776.513227890"
Oct 30 01:28:55.008332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:55.008259    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2" podStartSLOduration=2.008240935 podStartE2EDuration="2.008240935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:54.999295001 +0000 UTC m=+2776.546985498" watchObservedRunningTime="2026-10-30 01:28:55.008240935 +0000 UTC m=+2776.555931432"
Oct 30 01:28:55.540127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:55.540088    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"]
Oct 30 01:28:57.050671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:57.050635    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"]
Oct 30 01:28:57.065344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:57.065307    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f0263-m5xkk"]
Oct 30 01:28:57.088580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:57.088535    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:28:57.812375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:57.812327    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"]
Oct 30 01:28:57.812981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:57.812921    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" containerID="cri-o://e27347681759eb1a84bfa8ac5ad90731647c6da1ca7dd3fcea00eed0af95ec53" gracePeriod=30
Oct 30 01:28:58.609904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:58.609856    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe794d5-954f-4279-a5bd-51607de68f0a" path="/var/lib/kubelet/pods/4fe794d5-954f-4279-a5bd-51607de68f0a/volumes"
Oct 30 01:28:59.082596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.082514    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"]
Oct 30 01:28:59.115480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.111750    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcd394d-vgbrn"]
Oct 30 01:28:59.337526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.323405    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"]
Oct 30 01:28:59.337526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.323756    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy" containerID="cri-o://543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1" gracePeriod=30
Oct 30 01:28:59.436499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.434645    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"]
Oct 30 01:28:59.436499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.434838    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy" containerID="cri-o://05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631" gracePeriod=30
Oct 30 01:28:59.713501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.710995    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:28:59.713501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.711060    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:28:59.952856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.952810    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"]
Oct 30 01:28:59.954603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.954568    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerName="payload-pre-processing"
Oct 30 01:28:59.954804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.954788    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerName="payload-pre-processing"
Oct 30 01:28:59.955226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.955205    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb63edbc-f6c8-4581-9537-f3b49f54165e" containerName="payload-pre-processing"
Oct 30 01:28:59.988037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.987944    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"]
Oct 30 01:28:59.988319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.988286    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"]
Oct 30 01:28:59.997821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:28:59.997111    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.013811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.012536    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:00.033313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.029776    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:29:00.060522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.058671    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"]
Oct 30 01:29:00.060522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.058784    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.138832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.137507    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.138832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.137630    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.138832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.137663    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8jj\" (UniqueName: \"kubernetes.io/projected/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-kube-api-access-2r8jj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.138832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.137736    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.138832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.137802    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npb79\" (UniqueName: \"kubernetes.io/projected/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-kube-api-access-npb79\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.138832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.137991    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.142278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.142249    5104 generic.go:356] "Generic (PLEG): container finished" podID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerID="cc1ef63d8cbf4ba76e887d328e825e3386a7e13a47d86a08d4b1e605c802ceb9" exitCode=0
Oct 30 01:29:00.142492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.142329    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" event={"ID":"a3afcfed-ab8a-494d-a375-d5b88cf74fed","Type":"ContainerDied","Data":"cc1ef63d8cbf4ba76e887d328e825e3386a7e13a47d86a08d4b1e605c802ceb9"}
Oct 30 01:29:00.147226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.147200    5104 generic.go:356] "Generic (PLEG): container finished" podID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerID="3e656146389bc57a604490506a0f474ddf3f560281ea067ecc593e52785cefdd" exitCode=0
Oct 30 01:29:00.147390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.147376    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" event={"ID":"6f4f6143-8983-4709-bcc6-01c5def319d2","Type":"ContainerDied","Data":"3e656146389bc57a604490506a0f474ddf3f560281ea067ecc593e52785cefdd"}
Oct 30 01:29:00.147518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.147421    5104 scope.go:122] "RemoveContainer" containerID="3853c548357e366cb193ff58e5e4a1074d0dadedf64d2db788c80115f146ee93"
Oct 30 01:29:00.240332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.240250    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.240332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.240292    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8jj\" (UniqueName: \"kubernetes.io/projected/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-kube-api-access-2r8jj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.240612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.240365    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.240612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.240427    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npb79\" (UniqueName: \"kubernetes.io/projected/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-kube-api-access-npb79\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.240734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.240693    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.242004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.240876    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.242004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.241438    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.242004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.241596    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.245487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.244715    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.245487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.245247    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.249090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.249070    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:29:00.260010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.259985    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npb79\" (UniqueName: \"kubernetes.io/projected/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-kube-api-access-npb79\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.260226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.260201    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8jj\" (UniqueName: \"kubernetes.io/projected/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-kube-api-access-2r8jj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.272605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.272583    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:29:00.342025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.341989    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-envoy\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.342236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342146    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-certs\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342210    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-socket\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342254    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-credential-socket\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342288    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4f6143-8983-4709-bcc6-01c5def319d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f6143-8983-4709-bcc6-01c5def319d2-istiod-ca-cert\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342345    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-socket\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.342543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342384    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-podinfo\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342486    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-data\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342521    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-envoy\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342629    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-kube-api-access-n4xkd\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-kube-api-access-n4xkd\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.342706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342657    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-token\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.342706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342689    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istiod-ca-cert\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.342876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342732    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-podinfo\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.342876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342794    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-data\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.342876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342868    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-kube-api-access-rb9bp\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-kube-api-access-rb9bp\") pod \"6f4f6143-8983-4709-bcc6-01c5def319d2\" (UID: \"6f4f6143-8983-4709-bcc6-01c5def319d2\") "
Oct 30 01:29:00.343038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.342899    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-certs\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.343038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.343032    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-credential-socket\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.343147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.343081    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-token\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-token\") pod \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\" (UID: \"a3afcfed-ab8a-494d-a375-d5b88cf74fed\") "
Oct 30 01:29:00.343677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.343616    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-credential-socket" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.344518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.344447    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.344769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.344745    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-certs" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.345090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.345060    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istiod-ca-cert" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:00.345369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.345343    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-socket" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.345635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.345607    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-socket" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.345762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.345679    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4f6143-8983-4709-bcc6-01c5def319d2-istiod-ca-cert" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:00.346520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.346493    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-certs" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.346725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.346701    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-credential-socket" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.346952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.346922    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-data" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.347146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.347121    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-data" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.349906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.349878    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-envoy" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.350068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.349966    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-token" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:00.350068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.350048    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-podinfo" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:00.350473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.350417    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-envoy" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:00.350592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.350519    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-kube-api-access-rb9bp" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "kube-api-access-rb9bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:00.350592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.350556    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-token" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:00.350943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.350906    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-podinfo" pod "6f4f6143-8983-4709-bcc6-01c5def319d2" (UID: "6f4f6143-8983-4709-bcc6-01c5def319d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:00.352044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.352017    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-kube-api-access-n4xkd" pod "a3afcfed-ab8a-494d-a375-d5b88cf74fed" (UID: "a3afcfed-ab8a-494d-a375-d5b88cf74fed"). InnerVolumeSpecName "kube-api-access-n4xkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:00.368274 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.368243    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:00.442678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.442642    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462666    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462718    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462734    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462749    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f6143-8983-4709-bcc6-01c5def319d2-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462764    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462793    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462808    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462821    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462835    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4xkd\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-kube-api-access-n4xkd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462865    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462879    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462891    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462903    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462916    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9bp\" (UniqueName: \"kubernetes.io/projected/6f4f6143-8983-4709-bcc6-01c5def319d2-kube-api-access-rb9bp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462946    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462961    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afcfed-ab8a-494d-a375-d5b88cf74fed-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.465644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.462974    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afcfed-ab8a-494d-a375-d5b88cf74fed-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:00.542047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.541087    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t" podUID="64f12096-c6a9-4cb3-9ec6-a276d59f039b" containerName="payload-pre-processing" containerID="cri-o://1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02" gracePeriod=25
Oct 30 01:29:00.587870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.587729    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445248a7-9fa0-45c4-9278-579c204cc360" path="/var/lib/kubelet/pods/445248a7-9fa0-45c4-9278-579c204cc360/volumes"
Oct 30 01:29:00.700804 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:00.700762    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c92fb3_bef5_49c8_bb28_7e9b4edc3b79.slice/crio-b4eb6568d1334b26e759e88f294250091b8d8e5b109e02b3319eae88f7e1f226 WatchSource:0}: Error finding container b4eb6568d1334b26e759e88f294250091b8d8e5b109e02b3319eae88f7e1f226: Status 404 returned error can't find the container with id b4eb6568d1334b26e759e88f294250091b8d8e5b109e02b3319eae88f7e1f226
Oct 30 01:29:00.702726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.702537    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"]
Oct 30 01:29:00.824739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.824680    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"]
Oct 30 01:29:00.979051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:00.979023    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:29:01.002449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.002409    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f12096-c6a9-4cb3-9ec6-a276d59f039b-kube-api-access-bhkmc\" (UniqueName: \"kubernetes.io/projected/64f12096-c6a9-4cb3-9ec6-a276d59f039b-kube-api-access-bhkmc\") pod \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") "
Oct 30 01:29:01.002791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.002771    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f12096-c6a9-4cb3-9ec6-a276d59f039b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f12096-c6a9-4cb3-9ec6-a276d59f039b-plugins-config-volume\") pod \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\" (UID: \"64f12096-c6a9-4cb3-9ec6-a276d59f039b\") "
Oct 30 01:29:01.003222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.003193    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f12096-c6a9-4cb3-9ec6-a276d59f039b-plugins-config-volume" pod "64f12096-c6a9-4cb3-9ec6-a276d59f039b" (UID: "64f12096-c6a9-4cb3-9ec6-a276d59f039b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:01.003339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.003321    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f12096-c6a9-4cb3-9ec6-a276d59f039b-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:01.073886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.073809    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f12096-c6a9-4cb3-9ec6-a276d59f039b-kube-api-access-bhkmc" pod "64f12096-c6a9-4cb3-9ec6-a276d59f039b" (UID: "64f12096-c6a9-4cb3-9ec6-a276d59f039b"). InnerVolumeSpecName "kube-api-access-bhkmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:01.105385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.105350    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhkmc\" (UniqueName: \"kubernetes.io/projected/64f12096-c6a9-4cb3-9ec6-a276d59f039b-kube-api-access-bhkmc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:01.158285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.158246    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp" event={"ID":"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79","Type":"ContainerStarted","Data":"b4eb6568d1334b26e759e88f294250091b8d8e5b109e02b3319eae88f7e1f226"}
Oct 30 01:29:01.159771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.159749    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"
Oct 30 01:29:01.159900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.159781    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp" event={"ID":"a3afcfed-ab8a-494d-a375-d5b88cf74fed","Type":"ContainerDied","Data":"ec8383e05b424286c2273dcdb2a6229d4d1f011b97dc0c378a23bdcaa3d33f55"}
Oct 30 01:29:01.159900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.159847    5104 scope.go:122] "RemoveContainer" containerID="cc1ef63d8cbf4ba76e887d328e825e3386a7e13a47d86a08d4b1e605c802ceb9"
Oct 30 01:29:01.161412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.161388    5104 generic.go:356] "Generic (PLEG): container finished" podID="64f12096-c6a9-4cb3-9ec6-a276d59f039b" containerID="1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02" exitCode=0
Oct 30 01:29:01.161548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.161420    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t" event={"ID":"64f12096-c6a9-4cb3-9ec6-a276d59f039b","Type":"ContainerDied","Data":"1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02"}
Oct 30 01:29:01.161548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.161488    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t" event={"ID":"64f12096-c6a9-4cb3-9ec6-a276d59f039b","Type":"ContainerDied","Data":"84048f8debe9933562ec63172c1cb90ad31969c08c7de85d794338f821fbb6d8"}
Oct 30 01:29:01.161668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.161448    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"
Oct 30 01:29:01.163499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.163443    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"
Oct 30 01:29:01.163627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.163446    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh" event={"ID":"6f4f6143-8983-4709-bcc6-01c5def319d2","Type":"ContainerDied","Data":"00ce952889c5c4bb6eadb28d65aa539e026e823aff5b66316eff41f6598f858e"}
Oct 30 01:29:01.165418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.165392    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk" event={"ID":"21ffecf2-3ee2-4217-9072-e1cf151ea0e0","Type":"ContainerStarted","Data":"bfd5d1c384a70e78429c277a7746d8eb62a33cc16a2549037b06197c8975ff48"}
Oct 30 01:29:01.173465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.173429    5104 scope.go:122] "RemoveContainer" containerID="1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02"
Oct 30 01:29:01.195682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.195660    5104 scope.go:122] "RemoveContainer" containerID="1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02"
Oct 30 01:29:01.196051 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:01.196020    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02\": container with ID starting with 1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02 not found: ID does not exist" containerID="1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02"
Oct 30 01:29:01.196130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.196062    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02"} err="failed to get container status \"1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02\": rpc error: code = NotFound desc = could not find container \"1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02\": container with ID starting with 1ac56148e477171950d27f6638a3ecfc494e964fbba34c225c33de2a5e9b8f02 not found: ID does not exist"
Oct 30 01:29:01.196130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.196086    5104 scope.go:122] "RemoveContainer" containerID="3e656146389bc57a604490506a0f474ddf3f560281ea067ecc593e52785cefdd"
Oct 30 01:29:01.212733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.212701    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"]
Oct 30 01:29:01.217953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.217929    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-zqskh"]
Oct 30 01:29:01.250006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.249971    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"]
Oct 30 01:29:01.255039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.255009    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-64d664cfc9-sc42t"]
Oct 30 01:29:01.289768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.289743    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"]
Oct 30 01:29:01.296965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.296936    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-xftjp"]
Oct 30 01:29:01.378438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.378393    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fj9f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:29:01.378653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.378489    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:29:01.582628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.582593    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"]
Oct 30 01:29:01.584194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.584166    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy"
Oct 30 01:29:01.584397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.584380    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy"
Oct 30 01:29:01.584570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.584557    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy"
Oct 30 01:29:01.584666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.584655    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy"
Oct 30 01:29:01.584774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.584764    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy"
Oct 30 01:29:01.584854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.584844    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy"
Oct 30 01:29:01.585016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.585005    5104 cpu_manager.go:414] "containerMap: removing container" podUID="64f12096-c6a9-4cb3-9ec6-a276d59f039b" containerName="payload-pre-processing"
Oct 30 01:29:01.585106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.585085    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f12096-c6a9-4cb3-9ec6-a276d59f039b" containerName="payload-pre-processing"
Oct 30 01:29:01.585337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.585321    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f12096-c6a9-4cb3-9ec6-a276d59f039b" containerName="payload-pre-processing"
Oct 30 01:29:01.585405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.585342    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" containerName="istio-proxy"
Oct 30 01:29:01.585405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.585356    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy"
Oct 30 01:29:01.585405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.585370    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" containerName="istio-proxy"
Oct 30 01:29:01.590204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.590154    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.607012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.606982    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-679bd2-serving-cert\""
Oct 30 01:29:01.611327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.611294    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"]
Oct 30 01:29:01.614123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.614093    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tt95\" (UniqueName: \"kubernetes.io/projected/df745373-af7c-4f1f-8ada-df75bb8de3ad-kube-api-access-6tt95\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.614525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.614432    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.614600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.614542    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.614810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.614697    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df745373-af7c-4f1f-8ada-df75bb8de3ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.716262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.716223    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tt95\" (UniqueName: \"kubernetes.io/projected/df745373-af7c-4f1f-8ada-df75bb8de3ad-kube-api-access-6tt95\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.716437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.716357    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.716437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.716423    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.716615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.716541    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df745373-af7c-4f1f-8ada-df75bb8de3ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.717254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.717228    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df745373-af7c-4f1f-8ada-df75bb8de3ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.719175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.719151    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.719553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.719533    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.738277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.738242    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tt95\" (UniqueName: \"kubernetes.io/projected/df745373-af7c-4f1f-8ada-df75bb8de3ad-kube-api-access-6tt95\") pod \"maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:01.913510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:01.913478    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:02.180644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.180601    5104 generic.go:356] "Generic (PLEG): container finished" podID="21ffecf2-3ee2-4217-9072-e1cf151ea0e0" containerID="257619c1e40e0bcb9a6d3c1c273bd328f3c8f22ebeb29feff8a89b326d379a71" exitCode=0
Oct 30 01:29:02.180854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.180692    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk" event={"ID":"21ffecf2-3ee2-4217-9072-e1cf151ea0e0","Type":"ContainerDied","Data":"257619c1e40e0bcb9a6d3c1c273bd328f3c8f22ebeb29feff8a89b326d379a71"}
Oct 30 01:29:02.183190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.183160    5104 generic.go:356] "Generic (PLEG): container finished" podID="94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" containerID="b70b2c2a327ea7b9c2f80d4bc22da9b74b33e13060513d6549adaec5dbf25c84" exitCode=0
Oct 30 01:29:02.183345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.183242    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp" event={"ID":"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79","Type":"ContainerDied","Data":"b70b2c2a327ea7b9c2f80d4bc22da9b74b33e13060513d6549adaec5dbf25c84"}
Oct 30 01:29:02.232668 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:02.232634    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf745373_af7c_4f1f_8ada_df75bb8de3ad.slice/crio-f122c3c86a21dd6396a4f7b03d323dabf2384c8a6eda2eb6ae3274a03d696bc0 WatchSource:0}: Error finding container f122c3c86a21dd6396a4f7b03d323dabf2384c8a6eda2eb6ae3274a03d696bc0: Status 404 returned error can't find the container with id f122c3c86a21dd6396a4f7b03d323dabf2384c8a6eda2eb6ae3274a03d696bc0
Oct 30 01:29:02.233695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.233669    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"]
Oct 30 01:29:02.600004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.599917    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f12096-c6a9-4cb3-9ec6-a276d59f039b" path="/var/lib/kubelet/pods/64f12096-c6a9-4cb3-9ec6-a276d59f039b/volumes"
Oct 30 01:29:02.600587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.600569    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4f6143-8983-4709-bcc6-01c5def319d2" path="/var/lib/kubelet/pods/6f4f6143-8983-4709-bcc6-01c5def319d2/volumes"
Oct 30 01:29:02.601096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:02.601080    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3afcfed-ab8a-494d-a375-d5b88cf74fed" path="/var/lib/kubelet/pods/a3afcfed-ab8a-494d-a375-d5b88cf74fed/volumes"
Oct 30 01:29:03.293413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.293374    5104 generic.go:356] "Generic (PLEG): container finished" podID="5ca08310-2783-4282-be94-3d3a1da31f23" containerID="e27347681759eb1a84bfa8ac5ad90731647c6da1ca7dd3fcea00eed0af95ec53" exitCode=0
Oct 30 01:29:03.294205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.294175    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" event={"ID":"5ca08310-2783-4282-be94-3d3a1da31f23","Type":"ContainerDied","Data":"e27347681759eb1a84bfa8ac5ad90731647c6da1ca7dd3fcea00eed0af95ec53"}
Oct 30 01:29:03.298851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.298816    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm" event={"ID":"df745373-af7c-4f1f-8ada-df75bb8de3ad","Type":"ContainerStarted","Data":"6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81"}
Oct 30 01:29:03.302213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.302186    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm" event={"ID":"df745373-af7c-4f1f-8ada-df75bb8de3ad","Type":"ContainerStarted","Data":"f122c3c86a21dd6396a4f7b03d323dabf2384c8a6eda2eb6ae3274a03d696bc0"}
Oct 30 01:29:03.302538 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.302384    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:03.341807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.341448    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm" podStartSLOduration=1.949153966 podStartE2EDuration="2.341427539s" totalImagesPullingTime="392.273573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:03.338383552 +0000 UTC m=+2784.886074046" watchObservedRunningTime="2026-10-30 01:29:03.341427539 +0000 UTC m=+2784.889118038"
Oct 30 01:29:03.621141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.618532    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"]
Oct 30 01:29:03.621141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.618829    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy" containerID="cri-o://fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028" gracePeriod=30
Oct 30 01:29:03.659407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.659368    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:03.891561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.886649    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-kube-api-access-npb79\" (UniqueName: \"kubernetes.io/projected/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-kube-api-access-npb79\") pod \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") "
Oct 30 01:29:03.891561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.886742    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-xks-ca\") pod \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") "
Oct 30 01:29:03.891561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.886883    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-openshift-ca\") pod \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\" (UID: \"21ffecf2-3ee2-4217-9072-e1cf151ea0e0\") "
Oct 30 01:29:03.896213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.895761    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-openshift-ca" pod "21ffecf2-3ee2-4217-9072-e1cf151ea0e0" (UID: "21ffecf2-3ee2-4217-9072-e1cf151ea0e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:03.970921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.970851    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-kube-api-access-npb79" pod "21ffecf2-3ee2-4217-9072-e1cf151ea0e0" (UID: "21ffecf2-3ee2-4217-9072-e1cf151ea0e0"). InnerVolumeSpecName "kube-api-access-npb79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:03.971500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.971418    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:04.010500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:03.984811    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 01:29:04.025650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.011040    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npb79\" (UniqueName: \"kubernetes.io/projected/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-kube-api-access-npb79\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.025650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.011078    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.077716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.077685    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"]
Oct 30 01:29:04.079026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.078993    5104 cpu_manager.go:414] "containerMap: removing container" podUID="21ffecf2-3ee2-4217-9072-e1cf151ea0e0" containerName="revoke-keys"
Oct 30 01:29:04.080227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080205    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ffecf2-3ee2-4217-9072-e1cf151ea0e0" containerName="revoke-keys"
Oct 30 01:29:04.080394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080383    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy"
Oct 30 01:29:04.080487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080477    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy"
Oct 30 01:29:04.080559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080552    5104 cpu_manager.go:414] "containerMap: removing container" podUID="94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" containerName="revoke-keys"
Oct 30 01:29:04.080623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080616    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" containerName="revoke-keys"
Oct 30 01:29:04.080835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080826    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ffecf2-3ee2-4217-9072-e1cf151ea0e0" containerName="revoke-keys"
Oct 30 01:29:04.080907 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080900    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" containerName="istio-proxy"
Oct 30 01:29:04.080965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.080959    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" containerName="revoke-keys"
Oct 30 01:29:04.087508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.085094    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-xks-ca" pod "21ffecf2-3ee2-4217-9072-e1cf151ea0e0" (UID: "21ffecf2-3ee2-4217-9072-e1cf151ea0e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:04.093606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.093582    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.106134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.105379    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"]
Oct 30 01:29:04.116149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.116112    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-credential-socket\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.116366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.116345    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-credential-socket" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.116521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.116408    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-certs\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.116685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.116666    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-certs" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.116878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.116857    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca08310-2783-4282-be94-3d3a1da31f23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca08310-2783-4282-be94-3d3a1da31f23-istiod-ca-cert\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.117055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.117039    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-xks-ca\") pod \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") "
Oct 30 01:29:04.117178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.117163    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ca08310-2783-4282-be94-3d3a1da31f23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca08310-2783-4282-be94-3d3a1da31f23-istio-podinfo\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.117347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.117331    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-data\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.117566 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.117549    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-istio-token\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-istio-token\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.117784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.117756    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-kube-api-access-2r8jj\" (UniqueName: \"kubernetes.io/projected/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-kube-api-access-2r8jj\") pod \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") "
Oct 30 01:29:04.117966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.117948    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-socket\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.119411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.118078    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-kube-api-access-st8wg\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-kube-api-access-st8wg\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.119411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.118181    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-envoy\") pod \"5ca08310-2783-4282-be94-3d3a1da31f23\" (UID: \"5ca08310-2783-4282-be94-3d3a1da31f23\") "
Oct 30 01:29:04.119411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.118248    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-openshift-ca\") pod \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\" (UID: \"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79\") "
Oct 30 01:29:04.119411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.119169    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.119411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.119195    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.119411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.119209    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ffecf2-3ee2-4217-9072-e1cf151ea0e0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.126005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.117620    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca08310-2783-4282-be94-3d3a1da31f23-istiod-ca-cert" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:04.126005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.121134    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-openshift-ca" pod "94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" (UID: "94c92fb3-bef5-49c8-bb28-7e9b4edc3b79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:04.126005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.125279    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-socket" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.126005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.125745    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-data" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.129795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.129624    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-kube-api-access-2r8jj" pod "94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" (UID: "94c92fb3-bef5-49c8-bb28-7e9b4edc3b79"). InnerVolumeSpecName "kube-api-access-2r8jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:04.130196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.130162    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-xks-ca" pod "94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" (UID: "94c92fb3-bef5-49c8-bb28-7e9b4edc3b79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:04.131196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.131163    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-envoy" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.133118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.133091    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ca08310-2783-4282-be94-3d3a1da31f23-istio-podinfo" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:04.133381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.133335    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-kube-api-access-st8wg" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "kube-api-access-st8wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:04.135149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.135123    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-istio-token" pod "5ca08310-2783-4282-be94-3d3a1da31f23" (UID: "5ca08310-2783-4282-be94-3d3a1da31f23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:04.221369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.221321    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.221785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.221755    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcdr\" (UniqueName: \"kubernetes.io/projected/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-kube-api-access-bwcdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.221839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.221809    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.222105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222083    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222111    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222126    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca08310-2783-4282-be94-3d3a1da31f23-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222141    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222154    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca08310-2783-4282-be94-3d3a1da31f23-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222166    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222179    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222193    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8jj\" (UniqueName: \"kubernetes.io/projected/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79-kube-api-access-2r8jj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222205    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca08310-2783-4282-be94-3d3a1da31f23-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.222511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.222219    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st8wg\" (UniqueName: \"kubernetes.io/projected/5ca08310-2783-4282-be94-3d3a1da31f23-kube-api-access-st8wg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.313028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.312994    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"
Oct 30 01:29:04.313597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.313042    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp" event={"ID":"94c92fb3-bef5-49c8-bb28-7e9b4edc3b79","Type":"ContainerDied","Data":"b4eb6568d1334b26e759e88f294250091b8d8e5b109e02b3319eae88f7e1f226"}
Oct 30 01:29:04.313597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.313093    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4eb6568d1334b26e759e88f294250091b8d8e5b109e02b3319eae88f7e1f226"
Oct 30 01:29:04.315272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.315248    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"
Oct 30 01:29:04.315399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.315286    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6" event={"ID":"5ca08310-2783-4282-be94-3d3a1da31f23","Type":"ContainerDied","Data":"17aa689439ca1ad57ed346ebd77123082ece485a7a998860766cf460d8e84c26"}
Oct 30 01:29:04.315399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.315358    5104 scope.go:122] "RemoveContainer" containerID="e27347681759eb1a84bfa8ac5ad90731647c6da1ca7dd3fcea00eed0af95ec53"
Oct 30 01:29:04.324517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.324093    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"
Oct 30 01:29:04.324517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.324122    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk" event={"ID":"21ffecf2-3ee2-4217-9072-e1cf151ea0e0","Type":"ContainerDied","Data":"bfd5d1c384a70e78429c277a7746d8eb62a33cc16a2549037b06197c8975ff48"}
Oct 30 01:29:04.324517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.324179    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfd5d1c384a70e78429c277a7746d8eb62a33cc16a2549037b06197c8975ff48"
Oct 30 01:29:04.325366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.325230    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcdr\" (UniqueName: \"kubernetes.io/projected/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-kube-api-access-bwcdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.325635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.325326    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.326319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.325810    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.326319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.326127    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.329056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.329009    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.344605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.344580    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcdr\" (UniqueName: \"kubernetes.io/projected/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-kube-api-access-bwcdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.391224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.388267    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"]
Oct 30 01:29:04.394778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.394751    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fj9f6"]
Oct 30 01:29:04.417905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.417815    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:04.610659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.610618    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca08310-2783-4282-be94-3d3a1da31f23" path="/var/lib/kubelet/pods/5ca08310-2783-4282-be94-3d3a1da31f23/volumes"
Oct 30 01:29:04.741386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.741355    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"]
Oct 30 01:29:04.755041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.755012    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:29:04.783800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.783331    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:29:04.783800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.783384    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:29:04.830532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.827712    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" podUID="4054d583-1459-402e-a08f-477bdd2e067a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:29:04.835989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.835955    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-kube-api-access-2pdg5\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-kube-api-access-2pdg5\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.836148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836031    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-certs\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.836221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836190    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-credential-socket\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.836350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836278    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-socket\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.836439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836369    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-envoy\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.836439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836420    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-podinfo\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.836620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836563    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istiod-ca-cert\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.836682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836655    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-token\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-token\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.838030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836726    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-certs" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.838030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.836750    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-data\") pod \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\" (UID: \"d39f2e47-e349-4f0d-b7ec-9ef1891888ec\") "
Oct 30 01:29:04.838030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.837107    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-credential-socket" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.838030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.837622    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istiod-ca-cert" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:04.838030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.837752    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.838030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.837777    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.838030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.837994    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-socket" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.838899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.838875    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-data" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.844116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.842732    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-envoy" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:04.844116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.842839    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-kube-api-access-2pdg5" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "kube-api-access-2pdg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:04.844901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.844722    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-podinfo" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:04.845350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.845314    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-token" pod "d39f2e47-e349-4f0d-b7ec-9ef1891888ec" (UID: "d39f2e47-e349-4f0d-b7ec-9ef1891888ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:04.906615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.906585    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:29:04.939322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.939243    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.939322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.939275    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.939322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.939286    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.939322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.939294    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.939322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.939325    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.939692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.939334    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.939692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:04.939342    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdg5\" (UniqueName: \"kubernetes.io/projected/d39f2e47-e349-4f0d-b7ec-9ef1891888ec-kube-api-access-2pdg5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.040941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.040897    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-token\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-token\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041009    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-podinfo\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041072    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-certs\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041167    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-data\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041234    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-socket\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041300    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-credential-socket\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041435    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-kube-api-access-gtcm4\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-kube-api-access-gtcm4\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041506    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istiod-ca-cert\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041557    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-envoy\") pod \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\" (UID: \"530a2744-a0ae-46dd-a4c7-ada0d33538d4\") "
Oct 30 01:29:05.041781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041657    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-data" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:05.041911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041889    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-certs" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:05.041985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.041914    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-credential-socket" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:05.042254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.042222    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istiod-ca-cert" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:05.042384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.042363    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-socket" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:05.042662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.042636    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.042778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.042666    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.042778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.042680    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.042778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.042694    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.042778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.042709    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.044346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.044323    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-podinfo" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:05.044465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.044358    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-token" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:05.044465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.044403    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-kube-api-access-gtcm4" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "kube-api-access-gtcm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:05.044558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.044505    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-envoy" pod "530a2744-a0ae-46dd-a4c7-ada0d33538d4" (UID: "530a2744-a0ae-46dd-a4c7-ada0d33538d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:05.144652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.144615    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.144652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.144647    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.144652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.144657    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530a2744-a0ae-46dd-a4c7-ada0d33538d4-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.144905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.144666    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtcm4\" (UniqueName: \"kubernetes.io/projected/530a2744-a0ae-46dd-a4c7-ada0d33538d4-kube-api-access-gtcm4\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:05.334706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.334671    5104 generic.go:356] "Generic (PLEG): container finished" podID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerID="05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631" exitCode=0
Oct 30 01:29:05.335160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.334721    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"
Oct 30 01:29:05.335160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.334748    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" event={"ID":"530a2744-a0ae-46dd-a4c7-ada0d33538d4","Type":"ContainerDied","Data":"05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631"}
Oct 30 01:29:05.335160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.334839    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5" event={"ID":"530a2744-a0ae-46dd-a4c7-ada0d33538d4","Type":"ContainerDied","Data":"f6eec01cd4b26243d200ddced5f58cfd2f6215f19347d6f81cadde2867ccf460"}
Oct 30 01:29:05.335160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.334869    5104 scope.go:122] "RemoveContainer" containerID="05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631"
Oct 30 01:29:05.336891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.336861    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p" event={"ID":"d91d8abc-9f1c-40aa-b91e-c153a95f5df2","Type":"ContainerStarted","Data":"11f4f92ef6d9ef98b67827d967b6fdc9d7a80f411732480043cdb793b2e7e52f"}
Oct 30 01:29:05.337029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.336915    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p" event={"ID":"d91d8abc-9f1c-40aa-b91e-c153a95f5df2","Type":"ContainerStarted","Data":"6e987e2bf029153caf5fa193cb625e8857b39ed873a3d7fcd9be643f43ab7f3d"}
Oct 30 01:29:05.338838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.338815    5104 generic.go:356] "Generic (PLEG): container finished" podID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerID="543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1" exitCode=0
Oct 30 01:29:05.338992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.338855    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"
Oct 30 01:29:05.338992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.338892    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" event={"ID":"d39f2e47-e349-4f0d-b7ec-9ef1891888ec","Type":"ContainerDied","Data":"543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1"}
Oct 30 01:29:05.338992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.338949    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72" event={"ID":"d39f2e47-e349-4f0d-b7ec-9ef1891888ec","Type":"ContainerDied","Data":"ada1188a60577b476d94c13b0c271fff9040065a25baf4bd3984c86ae770a453"}
Oct 30 01:29:05.350438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.349249    5104 scope.go:122] "RemoveContainer" containerID="058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b"
Oct 30 01:29:05.364140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.364117    5104 scope.go:122] "RemoveContainer" containerID="05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631"
Oct 30 01:29:05.364469 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:05.364434    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631\": container with ID starting with 05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631 not found: ID does not exist" containerID="05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631"
Oct 30 01:29:05.364575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.364483    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631"} err="failed to get container status \"05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631\": rpc error: code = NotFound desc = could not find container \"05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631\": container with ID starting with 05fbbf41d03a850a6ebc0c1191546a0de0a6fcd81c9b844423182903fb283631 not found: ID does not exist"
Oct 30 01:29:05.364575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.364504    5104 scope.go:122] "RemoveContainer" containerID="058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b"
Oct 30 01:29:05.364798 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:05.364769    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b\": container with ID starting with 058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b not found: ID does not exist" containerID="058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b"
Oct 30 01:29:05.364872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.364804    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b"} err="failed to get container status \"058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b\": rpc error: code = NotFound desc = could not find container \"058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b\": container with ID starting with 058541311f1a7c146bbd1a3e2eb14c9549f8e50d0645ecc517dc48a0d7a1718b not found: ID does not exist"
Oct 30 01:29:05.364872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.364819    5104 scope.go:122] "RemoveContainer" containerID="543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1"
Oct 30 01:29:05.407009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.404192    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p" podStartSLOduration=1.404166513 podStartE2EDuration="1.404166513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:05.39011351 +0000 UTC m=+2786.937804007" watchObservedRunningTime="2026-10-30 01:29:05.404166513 +0000 UTC m=+2786.951857009"
Oct 30 01:29:05.411510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.411441    5104 scope.go:122] "RemoveContainer" containerID="543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1"
Oct 30 01:29:05.413489 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:05.412862    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1\": container with ID starting with 543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1 not found: ID does not exist" containerID="543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1"
Oct 30 01:29:05.413489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.412900    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1"} err="failed to get container status \"543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1\": rpc error: code = NotFound desc = could not find container \"543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1\": container with ID starting with 543e4267e3d34b21b53376d336d0eac45db17f31d8b9f5d2ad6cd6b89ea41ec1 not found: ID does not exist"
Oct 30 01:29:05.468123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.468086    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"]
Oct 30 01:29:05.475513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.473854    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-76z72"]
Oct 30 01:29:05.479180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.479154    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"]
Oct 30 01:29:05.484221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.483539    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-252z5"]
Oct 30 01:29:05.946169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.946141    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:29:05.946331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:05.946219    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:29:06.352868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:06.352834    5104 generic.go:356] "Generic (PLEG): container finished" podID="d91d8abc-9f1c-40aa-b91e-c153a95f5df2" containerID="11f4f92ef6d9ef98b67827d967b6fdc9d7a80f411732480043cdb793b2e7e52f" exitCode=0
Oct 30 01:29:06.353263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:06.352919    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p" event={"ID":"d91d8abc-9f1c-40aa-b91e-c153a95f5df2","Type":"ContainerDied","Data":"11f4f92ef6d9ef98b67827d967b6fdc9d7a80f411732480043cdb793b2e7e52f"}
Oct 30 01:29:06.582052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:06.582017    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" path="/var/lib/kubelet/pods/530a2744-a0ae-46dd-a4c7-ada0d33538d4/volumes"
Oct 30 01:29:06.582713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:06.582696    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" path="/var/lib/kubelet/pods/d39f2e47-e349-4f0d-b7ec-9ef1891888ec/volumes"
Oct 30 01:29:07.229468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.229427    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:29:07.269543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.269513    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:29:07.365867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.365777    5104 generic.go:356] "Generic (PLEG): container finished" podID="4054d583-1459-402e-a08f-477bdd2e067a" containerID="2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b" exitCode=0
Oct 30 01:29:07.366329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.365872    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"
Oct 30 01:29:07.366329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.365919    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" event={"ID":"4054d583-1459-402e-a08f-477bdd2e067a","Type":"ContainerDied","Data":"2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b"}
Oct 30 01:29:07.366329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.365984    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6" event={"ID":"4054d583-1459-402e-a08f-477bdd2e067a","Type":"ContainerDied","Data":"224c9f3b10f42d11fb7e44584e8d6cc3fbfcfd835f72a55e485e3bb839bce600"}
Oct 30 01:29:07.366329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.366021    5104 scope.go:122] "RemoveContainer" containerID="2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b"
Oct 30 01:29:07.367932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.367907    5104 generic.go:356] "Generic (PLEG): container finished" podID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerID="18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82" exitCode=0
Oct 30 01:29:07.368048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.367957    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"
Oct 30 01:29:07.368048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.367994    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" event={"ID":"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a","Type":"ContainerDied","Data":"18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82"}
Oct 30 01:29:07.368048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.368040    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" event={"ID":"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a","Type":"ContainerDied","Data":"32e4937b0a6fbc8a875e446b7e7c22a023ff3ff448dd62aade65b981a6d3b3db"}
Oct 30 01:29:07.378341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.378311    5104 scope.go:122] "RemoveContainer" containerID="2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b"
Oct 30 01:29:07.378988 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:07.378954    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b\": container with ID starting with 2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b not found: ID does not exist" containerID="2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b"
Oct 30 01:29:07.379111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.379000    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b"} err="failed to get container status \"2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b\": rpc error: code = NotFound desc = could not find container \"2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b\": container with ID starting with 2ecf7708478b56f97dfd3e4f644a70a636bf66985af9267b7fa3e2f1b199570b not found: ID does not exist"
Oct 30 01:29:07.379111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.379028    5104 scope.go:122] "RemoveContainer" containerID="18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82"
Oct 30 01:29:07.392564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.392531    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-plugins-config-volume\") pod \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") "
Oct 30 01:29:07.392754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.392700    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4054d583-1459-402e-a08f-477bdd2e067a-kube-api-access-gnzxt\" (UniqueName: \"kubernetes.io/projected/4054d583-1459-402e-a08f-477bdd2e067a-kube-api-access-gnzxt\") pod \"4054d583-1459-402e-a08f-477bdd2e067a\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") "
Oct 30 01:29:07.392754 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.392740    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4054d583-1459-402e-a08f-477bdd2e067a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4054d583-1459-402e-a08f-477bdd2e067a-plugins-config-volume\") pod \"4054d583-1459-402e-a08f-477bdd2e067a\" (UID: \"4054d583-1459-402e-a08f-477bdd2e067a\") "
Oct 30 01:29:07.392881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.392805    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-kube-api-access-lb7jw\" (UniqueName: \"kubernetes.io/projected/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-kube-api-access-lb7jw\") pod \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\" (UID: \"a74bf26f-2b9a-4d2f-8718-33b237dc7d4a\") "
Oct 30 01:29:07.393955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.393629    5104 scope.go:122] "RemoveContainer" containerID="18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82"
Oct 30 01:29:07.394293 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:07.394112    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82\": container with ID starting with 18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82 not found: ID does not exist" containerID="18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82"
Oct 30 01:29:07.394293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.394151    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4054d583-1459-402e-a08f-477bdd2e067a-plugins-config-volume" pod "4054d583-1459-402e-a08f-477bdd2e067a" (UID: "4054d583-1459-402e-a08f-477bdd2e067a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:07.394293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.394155    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82"} err="failed to get container status \"18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82\": rpc error: code = NotFound desc = could not find container \"18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82\": container with ID starting with 18a42ce273bcdc492e65bc68e6f09a413b178043e602fefdec0c5442fe499e82 not found: ID does not exist"
Oct 30 01:29:07.394293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.394261    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-plugins-config-volume" pod "a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" (UID: "a74bf26f-2b9a-4d2f-8718-33b237dc7d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:07.396009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.395982    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4054d583-1459-402e-a08f-477bdd2e067a-kube-api-access-gnzxt" pod "4054d583-1459-402e-a08f-477bdd2e067a" (UID: "4054d583-1459-402e-a08f-477bdd2e067a"). InnerVolumeSpecName "kube-api-access-gnzxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:07.396390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.396365    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-kube-api-access-lb7jw" pod "a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" (UID: "a74bf26f-2b9a-4d2f-8718-33b237dc7d4a"). InnerVolumeSpecName "kube-api-access-lb7jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:07.495832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.495735    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7jw\" (UniqueName: \"kubernetes.io/projected/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-kube-api-access-lb7jw\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:07.495832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.495770    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:07.495832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.495785    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnzxt\" (UniqueName: \"kubernetes.io/projected/4054d583-1459-402e-a08f-477bdd2e067a-kube-api-access-gnzxt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:07.495832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.495799    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4054d583-1459-402e-a08f-477bdd2e067a-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:07.530429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.530398    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:07.698205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.698170    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-xks-ca\") pod \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") "
Oct 30 01:29:07.698391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.698270    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-kube-api-access-bwcdr\" (UniqueName: \"kubernetes.io/projected/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-kube-api-access-bwcdr\") pod \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") "
Oct 30 01:29:07.698518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.698437    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-openshift-ca\") pod \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\" (UID: \"d91d8abc-9f1c-40aa-b91e-c153a95f5df2\") "
Oct 30 01:29:07.698791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.698753    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-openshift-ca" pod "d91d8abc-9f1c-40aa-b91e-c153a95f5df2" (UID: "d91d8abc-9f1c-40aa-b91e-c153a95f5df2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:07.699299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.699269    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:07.700786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.700760    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-kube-api-access-bwcdr" pod "d91d8abc-9f1c-40aa-b91e-c153a95f5df2" (UID: "d91d8abc-9f1c-40aa-b91e-c153a95f5df2"). InnerVolumeSpecName "kube-api-access-bwcdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:07.700902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.700889    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-xks-ca" pod "d91d8abc-9f1c-40aa-b91e-c153a95f5df2" (UID: "d91d8abc-9f1c-40aa-b91e-c153a95f5df2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:07.721681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.721644    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"]
Oct 30 01:29:07.726103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.726076    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2be695-7574689bb7-pf5f6"]
Oct 30 01:29:07.753994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.753962    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"]
Oct 30 01:29:07.759447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.759415    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2"]
Oct 30 01:29:07.800944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.800901    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:07.800944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.800936    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwcdr\" (UniqueName: \"kubernetes.io/projected/d91d8abc-9f1c-40aa-b91e-c153a95f5df2-kube-api-access-bwcdr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:07.883899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:07.883864    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:29:08.108635 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:08.108542    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2be695-55dcd95b68-qdsx2" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.168:50052\" within 1s: context deadline exceeded"
Oct 30 01:29:08.108635 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:08.108604    5104 logging.go:55] [core] [Channel #14051 SubChannel #14052]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.168:50052", ServerName: "10.132.0.168:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.168:50052: operation was canceled"
Oct 30 01:29:08.382745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:08.382647    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"
Oct 30 01:29:08.382745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:08.382646    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p" event={"ID":"d91d8abc-9f1c-40aa-b91e-c153a95f5df2","Type":"ContainerDied","Data":"6e987e2bf029153caf5fa193cb625e8857b39ed873a3d7fcd9be643f43ab7f3d"}
Oct 30 01:29:08.383299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:08.382825    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e987e2bf029153caf5fa193cb625e8857b39ed873a3d7fcd9be643f43ab7f3d"
Oct 30 01:29:08.582061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:08.582025    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4054d583-1459-402e-a08f-477bdd2e067a" path="/var/lib/kubelet/pods/4054d583-1459-402e-a08f-477bdd2e067a/volumes"
Oct 30 01:29:08.582550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:08.582531    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" path="/var/lib/kubelet/pods/a74bf26f-2b9a-4d2f-8718-33b237dc7d4a/volumes"
Oct 30 01:29:08.941250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:08.941218    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:29:09.025837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.025801    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b131465-6995-4c54-909b-49205c47f220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b131465-6995-4c54-909b-49205c47f220-istio-podinfo\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.026032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.025851    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-kube-api-access-7kc5c\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-kube-api-access-7kc5c\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.026032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.025935    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-data\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.026032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.025977    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-socket\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.026032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026010    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-envoy\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.026307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026116    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b131465-6995-4c54-909b-49205c47f220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b131465-6995-4c54-909b-49205c47f220-istiod-ca-cert\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.026307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026221    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-credential-socket\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.026307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026242    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-data" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026307    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-istio-token\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-istio-token\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026372    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-certs\") pod \"1b131465-6995-4c54-909b-49205c47f220\" (UID: \"1b131465-6995-4c54-909b-49205c47f220\") "
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026626    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-credential-socket" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026881    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b131465-6995-4c54-909b-49205c47f220-istiod-ca-cert" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.026887    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-certs" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.027122    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-socket" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.027433    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.027477    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.027492    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.027508    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b131465-6995-4c54-909b-49205c47f220-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.028367 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.027520    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.030339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.030300    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b131465-6995-4c54-909b-49205c47f220-istio-podinfo" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:09.030844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.030750    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-envoy" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:09.030844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.030789    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-kube-api-access-7kc5c" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "kube-api-access-7kc5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:09.031096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.030992    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-istio-token" pod "1b131465-6995-4c54-909b-49205c47f220" (UID: "1b131465-6995-4c54-909b-49205c47f220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:09.106916 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:09.106880    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:29:09.129120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.129085    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b131465-6995-4c54-909b-49205c47f220-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.129120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.129116    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.129120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.129126    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b131465-6995-4c54-909b-49205c47f220-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.129366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.129135    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kc5c\" (UniqueName: \"kubernetes.io/projected/1b131465-6995-4c54-909b-49205c47f220-kube-api-access-7kc5c\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:09.395029 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.394995    5104 generic.go:356] "Generic (PLEG): container finished" podID="1b131465-6995-4c54-909b-49205c47f220" containerID="fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028" exitCode=0
Oct 30 01:29:09.395534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.395053    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"
Oct 30 01:29:09.395534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.395080    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" event={"ID":"1b131465-6995-4c54-909b-49205c47f220","Type":"ContainerDied","Data":"fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028"}
Oct 30 01:29:09.395534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.395152    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp" event={"ID":"1b131465-6995-4c54-909b-49205c47f220","Type":"ContainerDied","Data":"c5dccab87174a375efd557d19e7fe8daeafb71a3f74cbb31a93580e3bef65f2d"}
Oct 30 01:29:09.395534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.395188    5104 scope.go:122] "RemoveContainer" containerID="fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028"
Oct 30 01:29:09.417596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.417572    5104 scope.go:122] "RemoveContainer" containerID="ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1"
Oct 30 01:29:09.429375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.429350    5104 scope.go:122] "RemoveContainer" containerID="fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028"
Oct 30 01:29:09.429774 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:09.429749    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028\": container with ID starting with fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028 not found: ID does not exist" containerID="fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028"
Oct 30 01:29:09.429835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.429784    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028"} err="failed to get container status \"fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028\": rpc error: code = NotFound desc = could not find container \"fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028\": container with ID starting with fe79614ce81e412590e202da20e5c59c0f674ceacd9f509470171f1c7b6fa028 not found: ID does not exist"
Oct 30 01:29:09.429835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.429805    5104 scope.go:122] "RemoveContainer" containerID="ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1"
Oct 30 01:29:09.430125 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:09.430101    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1\": container with ID starting with ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1 not found: ID does not exist" containerID="ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1"
Oct 30 01:29:09.430166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.430136    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1"} err="failed to get container status \"ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1\": rpc error: code = NotFound desc = could not find container \"ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1\": container with ID starting with ec1bb716f03e8643a77e78c66b19122edb6296972bc7055d81d65c4d95aae5a1 not found: ID does not exist"
Oct 30 01:29:09.447289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.447256    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"]
Oct 30 01:29:09.458465 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:09.458415    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f0263-openshift-default-6c6d84745c-8zzqp"]
Oct 30 01:29:10.027601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:10.027565    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"]
Oct 30 01:29:10.031950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:10.031921    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee3b1-7npzv"]
Oct 30 01:29:10.347704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:10.347619    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:29:10.438057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:10.438019    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"]
Oct 30 01:29:10.470836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:10.470801    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"]
Oct 30 01:29:10.604620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:10.604536    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b131465-6995-4c54-909b-49205c47f220" path="/var/lib/kubelet/pods/1b131465-6995-4c54-909b-49205c47f220/volumes"
Oct 30 01:29:10.605246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:10.605219    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c85f918-dfaa-41bf-96c3-d6a7095d1549" path="/var/lib/kubelet/pods/9c85f918-dfaa-41bf-96c3-d6a7095d1549/volumes"
Oct 30 01:29:12.346487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:12.341891    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"]
Oct 30 01:29:12.497520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:12.497434    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2be695-ndjn7"]
Oct 30 01:29:12.581956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:12.581921    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465bd99a-d8e3-4a68-a705-28f568b7b099" path="/var/lib/kubelet/pods/465bd99a-d8e3-4a68-a705-28f568b7b099/volumes"
Oct 30 01:29:14.514791 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:14.514746    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"]
Oct 30 01:29:14.546678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:14.546643    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"]
Oct 30 01:29:15.438576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:15.438533    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn" podUID="5bde04ee-fb54-4f94-9fa1-23381376ad6c" containerName="payload-processing" containerID="cri-o://1557eb8ae86a5ce7c7f6d0d066de9702d6a5c328ecc0fbeb35ca3414c16c5c78" gracePeriod=25
Oct 30 01:29:15.471613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:15.471560    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj" podUID="e437e532-1c76-4575-885a-01d40c1b4570" containerName="payload-pre-processing" containerID="cri-o://625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555" gracePeriod=25
Oct 30 01:29:15.570366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:15.570332    5104 generic.go:356] "Generic (PLEG): container finished" podID="5bde04ee-fb54-4f94-9fa1-23381376ad6c" containerID="1557eb8ae86a5ce7c7f6d0d066de9702d6a5c328ecc0fbeb35ca3414c16c5c78" exitCode=0
Oct 30 01:29:15.570840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:15.570402    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn" event={"ID":"5bde04ee-fb54-4f94-9fa1-23381376ad6c","Type":"ContainerDied","Data":"1557eb8ae86a5ce7c7f6d0d066de9702d6a5c328ecc0fbeb35ca3414c16c5c78"}
Oct 30 01:29:15.891576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:15.880330    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:29:15.973313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:15.973277    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:29:16.017980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.017945    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bde04ee-fb54-4f94-9fa1-23381376ad6c-kube-api-access-4ndd5\" (UniqueName: \"kubernetes.io/projected/5bde04ee-fb54-4f94-9fa1-23381376ad6c-kube-api-access-4ndd5\") pod \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") "
Oct 30 01:29:16.018224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.018031    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bde04ee-fb54-4f94-9fa1-23381376ad6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde04ee-fb54-4f94-9fa1-23381376ad6c-plugins-config-volume\") pod \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\" (UID: \"5bde04ee-fb54-4f94-9fa1-23381376ad6c\") "
Oct 30 01:29:16.018512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.018481    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bde04ee-fb54-4f94-9fa1-23381376ad6c-plugins-config-volume" pod "5bde04ee-fb54-4f94-9fa1-23381376ad6c" (UID: "5bde04ee-fb54-4f94-9fa1-23381376ad6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:16.018916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.018891    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde04ee-fb54-4f94-9fa1-23381376ad6c-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:16.020893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.020865    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bde04ee-fb54-4f94-9fa1-23381376ad6c-kube-api-access-4ndd5" pod "5bde04ee-fb54-4f94-9fa1-23381376ad6c" (UID: "5bde04ee-fb54-4f94-9fa1-23381376ad6c"). InnerVolumeSpecName "kube-api-access-4ndd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:16.119697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.119595    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e437e532-1c76-4575-885a-01d40c1b4570-kube-api-access-dt2xp\" (UniqueName: \"kubernetes.io/projected/e437e532-1c76-4575-885a-01d40c1b4570-kube-api-access-dt2xp\") pod \"e437e532-1c76-4575-885a-01d40c1b4570\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") "
Oct 30 01:29:16.119697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.119691    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e437e532-1c76-4575-885a-01d40c1b4570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e437e532-1c76-4575-885a-01d40c1b4570-plugins-config-volume\") pod \"e437e532-1c76-4575-885a-01d40c1b4570\" (UID: \"e437e532-1c76-4575-885a-01d40c1b4570\") "
Oct 30 01:29:16.120137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.120107    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e437e532-1c76-4575-885a-01d40c1b4570-plugins-config-volume" pod "e437e532-1c76-4575-885a-01d40c1b4570" (UID: "e437e532-1c76-4575-885a-01d40c1b4570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:16.120583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.120563    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e437e532-1c76-4575-885a-01d40c1b4570-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:16.120676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.120586    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ndd5\" (UniqueName: \"kubernetes.io/projected/5bde04ee-fb54-4f94-9fa1-23381376ad6c-kube-api-access-4ndd5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:16.121867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.121838    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e437e532-1c76-4575-885a-01d40c1b4570-kube-api-access-dt2xp" pod "e437e532-1c76-4575-885a-01d40c1b4570" (UID: "e437e532-1c76-4575-885a-01d40c1b4570"). InnerVolumeSpecName "kube-api-access-dt2xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:16.222096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.222043    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2xp\" (UniqueName: \"kubernetes.io/projected/e437e532-1c76-4575-885a-01d40c1b4570-kube-api-access-dt2xp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:16.582242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.582211    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"
Oct 30 01:29:16.583091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.583058    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn" event={"ID":"5bde04ee-fb54-4f94-9fa1-23381376ad6c","Type":"ContainerDied","Data":"bfa09bac01673398cbfe6bc5304a546ab34f882afcbf36e5330f69b1b4d56dfc"}
Oct 30 01:29:16.583302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.583283    5104 scope.go:122] "RemoveContainer" containerID="1557eb8ae86a5ce7c7f6d0d066de9702d6a5c328ecc0fbeb35ca3414c16c5c78"
Oct 30 01:29:16.584086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.584055    5104 generic.go:356] "Generic (PLEG): container finished" podID="e437e532-1c76-4575-885a-01d40c1b4570" containerID="625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555" exitCode=0
Oct 30 01:29:16.584276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.584256    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj" event={"ID":"e437e532-1c76-4575-885a-01d40c1b4570","Type":"ContainerDied","Data":"625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555"}
Oct 30 01:29:16.584428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.584302    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj" event={"ID":"e437e532-1c76-4575-885a-01d40c1b4570","Type":"ContainerDied","Data":"60be3dd85634d662fbd1d397b036d1cd95c1e5650f683d351af2a1f2d4e11a15"}
Oct 30 01:29:16.584428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.584391    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"
Oct 30 01:29:16.607591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.607570    5104 scope.go:122] "RemoveContainer" containerID="625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555"
Oct 30 01:29:16.620767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.620707    5104 scope.go:122] "RemoveContainer" containerID="625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555"
Oct 30 01:29:16.621070 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:16.621048    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555\": container with ID starting with 625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555 not found: ID does not exist" containerID="625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555"
Oct 30 01:29:16.621154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.621082    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555"} err="failed to get container status \"625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555\": rpc error: code = NotFound desc = could not find container \"625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555\": container with ID starting with 625204b522d08ec516980959c9f6a87dc63e62be8ab1d0f18608e25e668f7555 not found: ID does not exist"
Oct 30 01:29:16.648005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.647974    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"]
Oct 30 01:29:16.659210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.659179    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-999ff9-757d5774cd-rmjfn"]
Oct 30 01:29:16.692761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.692711    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"]
Oct 30 01:29:16.696896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:16.696862    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-999ff9-867b845c5f-cg2gj"]
Oct 30 01:29:18.700480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:18.691920    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bde04ee-fb54-4f94-9fa1-23381376ad6c" path="/var/lib/kubelet/pods/5bde04ee-fb54-4f94-9fa1-23381376ad6c/volumes"
Oct 30 01:29:18.700480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:18.692674    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e437e532-1c76-4575-885a-01d40c1b4570" path="/var/lib/kubelet/pods/e437e532-1c76-4575-885a-01d40c1b4570/volumes"
Oct 30 01:29:18.862964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:18.861701    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"]
Oct 30 01:29:18.862964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:18.861780    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"]
Oct 30 01:29:18.862964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:18.861975    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" containerID="cri-o://4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd" gracePeriod=30
Oct 30 01:29:18.862964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:18.862078    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" containerID="cri-o://d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9" gracePeriod=30
Oct 30 01:29:19.414007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.413970    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"]
Oct 30 01:29:19.414798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414773    5104 cpu_manager.go:414] "containerMap: removing container" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy"
Oct 30 01:29:19.414798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414791    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414813    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5bde04ee-fb54-4f94-9fa1-23381376ad6c" containerName="payload-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414818    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bde04ee-fb54-4f94-9fa1-23381376ad6c" containerName="payload-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414827    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4054d583-1459-402e-a08f-477bdd2e067a" containerName="payload-pre-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414834    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4054d583-1459-402e-a08f-477bdd2e067a" containerName="payload-pre-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414851    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e437e532-1c76-4575-885a-01d40c1b4570" containerName="payload-pre-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414856    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e437e532-1c76-4575-885a-01d40c1b4570" containerName="payload-pre-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414864    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d91d8abc-9f1c-40aa-b91e-c153a95f5df2" containerName="revoke-keys"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414868    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91d8abc-9f1c-40aa-b91e-c153a95f5df2" containerName="revoke-keys"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414874    5104 cpu_manager.go:414] "containerMap: removing container" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414878    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414887    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414892    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414906    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerName="payload-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414911    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerName="payload-processing"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414924    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414928    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414946    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy"
Oct 30 01:29:19.414977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.414951    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415063    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39f2e47-e349-4f0d-b7ec-9ef1891888ec" containerName="istio-proxy"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415073    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415080    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bde04ee-fb54-4f94-9fa1-23381376ad6c" containerName="payload-processing"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415086    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4054d583-1459-402e-a08f-477bdd2e067a" containerName="payload-pre-processing"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415095    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e437e532-1c76-4575-885a-01d40c1b4570" containerName="payload-pre-processing"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415100    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91d8abc-9f1c-40aa-b91e-c153a95f5df2" containerName="revoke-keys"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415105    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="530a2744-a0ae-46dd-a4c7-ada0d33538d4" containerName="istio-proxy"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415111    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415117    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74bf26f-2b9a-4d2f-8718-33b237dc7d4a" containerName="payload-processing"
Oct 30 01:29:19.415609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.415123    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b131465-6995-4c54-909b-49205c47f220" containerName="istio-proxy"
Oct 30 01:29:19.419981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.419959    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.434010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.433981    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:29:19.434184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.434041    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:19.436281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.436252    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"]
Oct 30 01:29:19.486960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.486920    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrtg\" (UniqueName: \"kubernetes.io/projected/a29f43b8-6662-418c-94ea-c0be9254013f-kube-api-access-tkrtg\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.487177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.487041    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.487177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.487096    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.515938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.515892    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" podUID="f0b4d577-6ef2-427c-8bd9-527086752e20" containerName="payload-processing" containerID="cri-o://2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56" gracePeriod=25
Oct 30 01:29:19.547589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.547540    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" podUID="4905cca2-375e-46a5-9845-38ab8171ea14" containerName="payload-pre-processing" containerID="cri-o://b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772" gracePeriod=25
Oct 30 01:29:19.589196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.589158    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrtg\" (UniqueName: \"kubernetes.io/projected/a29f43b8-6662-418c-94ea-c0be9254013f-kube-api-access-tkrtg\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.589434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.589211    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.589434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.589237    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.590023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.589997    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.591714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.591686    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.622393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.622362    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrtg\" (UniqueName: \"kubernetes.io/projected/a29f43b8-6662-418c-94ea-c0be9254013f-kube-api-access-tkrtg\") pod \"maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:19.735845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:19.735807    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:20.041962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:20.041927    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"]
Oct 30 01:29:20.043485 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:20.043437    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29f43b8_6662_418c_94ea_c0be9254013f.slice/crio-c6a40bdf43e18a70936d06cbe5b4a2019a3b853b46c94e63c3bb8aec86d2d86a WatchSource:0}: Error finding container c6a40bdf43e18a70936d06cbe5b4a2019a3b853b46c94e63c3bb8aec86d2d86a: Status 404 returned error can't find the container with id c6a40bdf43e18a70936d06cbe5b4a2019a3b853b46c94e63c3bb8aec86d2d86a
Oct 30 01:29:20.667337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:20.667245    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv" event={"ID":"a29f43b8-6662-418c-94ea-c0be9254013f","Type":"ContainerStarted","Data":"582ef7131a32bc43c957f702b43424d760bf158a4b418f6d63a262185ac1936d"}
Oct 30 01:29:20.667337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:20.667305    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv" event={"ID":"a29f43b8-6662-418c-94ea-c0be9254013f","Type":"ContainerStarted","Data":"c6a40bdf43e18a70936d06cbe5b4a2019a3b853b46c94e63c3bb8aec86d2d86a"}
Oct 30 01:29:20.709137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:20.709082    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv" podStartSLOduration=1.709068026 podStartE2EDuration="1.709068026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:20.704399937 +0000 UTC m=+2802.252090433" watchObservedRunningTime="2026-10-30 01:29:20.709068026 +0000 UTC m=+2802.256758521"
Oct 30 01:29:20.719604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:20.719566    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:29:20.719803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:20.719700    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:29:21.503873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:21.503833    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"]
Oct 30 01:29:21.514882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:21.514844    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-64f472-drd7p"]
Oct 30 01:29:21.679293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:21.679255    5104 generic.go:356] "Generic (PLEG): container finished" podID="a29f43b8-6662-418c-94ea-c0be9254013f" containerID="582ef7131a32bc43c957f702b43424d760bf158a4b418f6d63a262185ac1936d" exitCode=0
Oct 30 01:29:21.679569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:21.679305    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv" event={"ID":"a29f43b8-6662-418c-94ea-c0be9254013f","Type":"ContainerDied","Data":"582ef7131a32bc43c957f702b43424d760bf158a4b418f6d63a262185ac1936d"}
Oct 30 01:29:22.579050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.579016    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91d8abc-9f1c-40aa-b91e-c153a95f5df2" path="/var/lib/kubelet/pods/d91d8abc-9f1c-40aa-b91e-c153a95f5df2/volumes"
Oct 30 01:29:22.869281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.869240    5104 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:29:22.869437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.869308    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:29:22.888913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.888879    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:22.965745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.965704    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29f43b8-6662-418c-94ea-c0be9254013f-kube-api-access-tkrtg\" (UniqueName: \"kubernetes.io/projected/a29f43b8-6662-418c-94ea-c0be9254013f-kube-api-access-tkrtg\") pod \"a29f43b8-6662-418c-94ea-c0be9254013f\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") "
Oct 30 01:29:22.965958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.965853    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-openshift-ca\") pod \"a29f43b8-6662-418c-94ea-c0be9254013f\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") "
Oct 30 01:29:22.965958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.965886    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-xks-ca\") pod \"a29f43b8-6662-418c-94ea-c0be9254013f\" (UID: \"a29f43b8-6662-418c-94ea-c0be9254013f\") "
Oct 30 01:29:22.966284 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.966254    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-openshift-ca" pod "a29f43b8-6662-418c-94ea-c0be9254013f" (UID: "a29f43b8-6662-418c-94ea-c0be9254013f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:22.966693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.966671    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:22.968136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.968111    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-xks-ca" pod "a29f43b8-6662-418c-94ea-c0be9254013f" (UID: "a29f43b8-6662-418c-94ea-c0be9254013f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:22.968136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:22.968129    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29f43b8-6662-418c-94ea-c0be9254013f-kube-api-access-tkrtg" pod "a29f43b8-6662-418c-94ea-c0be9254013f" (UID: "a29f43b8-6662-418c-94ea-c0be9254013f"). InnerVolumeSpecName "kube-api-access-tkrtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:23.067777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:23.067738    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkrtg\" (UniqueName: \"kubernetes.io/projected/a29f43b8-6662-418c-94ea-c0be9254013f-kube-api-access-tkrtg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:23.067777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:23.067770    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f43b8-6662-418c-94ea-c0be9254013f-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:23.704244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:23.704209    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"
Oct 30 01:29:23.704244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:23.704234    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv" event={"ID":"a29f43b8-6662-418c-94ea-c0be9254013f","Type":"ContainerDied","Data":"c6a40bdf43e18a70936d06cbe5b4a2019a3b853b46c94e63c3bb8aec86d2d86a"}
Oct 30 01:29:23.704761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:23.704292    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a40bdf43e18a70936d06cbe5b4a2019a3b853b46c94e63c3bb8aec86d2d86a"
Oct 30 01:29:24.214801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.214766    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:29:24.227564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.227534    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"]
Oct 30 01:29:24.238867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.238741    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"]
Oct 30 01:29:24.270268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.270234    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:29:24.287191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.287147    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istiod-ca-cert\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.287388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.287276    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-credential-socket\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.287388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.287328    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-podinfo\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.287388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.287385    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-data\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.287602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.287439    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-kube-api-access-px9q4\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.290388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.288540    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-credential-socket" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.290388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.288550    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-data" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.290388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.288840    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istiod-ca-cert" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:24.301811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.293747    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-token\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-token\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.301811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.295172    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-envoy\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.301811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.295266    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-socket\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.301811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.295367    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-certs\") pod \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\" (UID: \"e4ac2245-ed16-4744-b7c8-cc67c20b47b7\") "
Oct 30 01:29:24.306709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.305942    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-envoy" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.318801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.318139    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-podinfo" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:24.320600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.320562    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-kube-api-access-px9q4" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "kube-api-access-px9q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:24.322042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.322015    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-socket" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.327570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.322356    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-certs" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.327570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.322528    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-token" pod "e4ac2245-ed16-4744-b7c8-cc67c20b47b7" (UID: "e4ac2245-ed16-4744-b7c8-cc67c20b47b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:24.327570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.322914    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.327838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.327601    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.327838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.327625    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.327838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.327641    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.428116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428080    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-certs\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428135    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-data\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428218    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-socket\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428261    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-credential-socket\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428404    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-data" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.428515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428420    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-certs" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.428515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428448    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-kube-api-access-pn79n\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-kube-api-access-pn79n\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428532    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-podinfo\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428550    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-socket" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.428707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428582    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istiod-ca-cert\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428605    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-credential-socket" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.428707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428682    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-envoy\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.428707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428706    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-token\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-token\") pod \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\" (UID: \"dab5a5f5-36f7-4f21-979f-c013a3e65d2b\") "
Oct 30 01:29:24.429026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.428915    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istiod-ca-cert" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:24.429531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.429505    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.429973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.429942    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-kube-api-access-px9q4\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.429966    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.430195    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.430209    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.430217    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.430226    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.430237    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.430251    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.430261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.430263    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ac2245-ed16-4744-b7c8-cc67c20b47b7-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.431399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.431376    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-podinfo" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:24.431506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.431418    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-token" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:24.431580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.431509    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-kube-api-access-pn79n" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "kube-api-access-pn79n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:24.431728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.431712    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-envoy" pod "dab5a5f5-36f7-4f21-979f-c013a3e65d2b" (UID: "dab5a5f5-36f7-4f21-979f-c013a3e65d2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:24.531834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.531732    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn79n\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-kube-api-access-pn79n\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.531834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.531771    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.531834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.531785    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.531834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.531796    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab5a5f5-36f7-4f21-979f-c013a3e65d2b-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:24.716281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.716240    5104 generic.go:356] "Generic (PLEG): container finished" podID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerID="d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9" exitCode=0
Oct 30 01:29:24.716823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.716318    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"
Oct 30 01:29:24.716823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.716320    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" event={"ID":"e4ac2245-ed16-4744-b7c8-cc67c20b47b7","Type":"ContainerDied","Data":"d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9"}
Oct 30 01:29:24.716823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.716384    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj" event={"ID":"e4ac2245-ed16-4744-b7c8-cc67c20b47b7","Type":"ContainerDied","Data":"3268358efa9c7d74ac22c4a51ef33e5bfc88b3f4a29612bc4e292211ac0742db"}
Oct 30 01:29:24.716823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.716412    5104 scope.go:122] "RemoveContainer" containerID="d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9"
Oct 30 01:29:24.718498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.718433    5104 generic.go:356] "Generic (PLEG): container finished" podID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerID="4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd" exitCode=0
Oct 30 01:29:24.718624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.718518    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" event={"ID":"dab5a5f5-36f7-4f21-979f-c013a3e65d2b","Type":"ContainerDied","Data":"4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd"}
Oct 30 01:29:24.718624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.718595    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj" event={"ID":"dab5a5f5-36f7-4f21-979f-c013a3e65d2b","Type":"ContainerDied","Data":"84e9f4b6a746cb86b0976baca3b3a796fd9723d28b87942ff300dddf4746167d"}
Oct 30 01:29:24.718624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.718544    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"
Oct 30 01:29:24.732091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.732066    5104 scope.go:122] "RemoveContainer" containerID="d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9"
Oct 30 01:29:24.732828 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:24.732783    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9\": container with ID starting with d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9 not found: ID does not exist" containerID="d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9"
Oct 30 01:29:24.732976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.732829    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9"} err="failed to get container status \"d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9\": rpc error: code = NotFound desc = could not find container \"d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9\": container with ID starting with d564e33b44e6f45963fb9beec569497ecaf530f21549bbd3b55fb10f247947c9 not found: ID does not exist"
Oct 30 01:29:24.732976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.732858    5104 scope.go:122] "RemoveContainer" containerID="4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd"
Oct 30 01:29:24.989535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:24.978675    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"]
Oct 30 01:29:25.154092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.153685    5104 scope.go:122] "RemoveContainer" containerID="4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd"
Oct 30 01:29:25.154092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.154030    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-blbmj"]
Oct 30 01:29:25.195644 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:25.195143    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd\": container with ID starting with 4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd not found: ID does not exist" containerID="4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd"
Oct 30 01:29:25.195644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.195196    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd"} err="failed to get container status \"4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd\": rpc error: code = NotFound desc = could not find container \"4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd\": container with ID starting with 4ac9b18628a9d91be49acfd78d2c7751e364235485dc5fe07916f7bda0d845cd not found: ID does not exist"
Oct 30 01:29:25.220502 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.220465    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"]
Oct 30 01:29:25.241022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.240988    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-2cxjj"]
Oct 30 01:29:25.256599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.255325    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"]
Oct 30 01:29:25.256599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.256509    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy"
Oct 30 01:29:25.256599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.256532    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy"
Oct 30 01:29:25.259770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.256990    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a29f43b8-6662-418c-94ea-c0be9254013f" containerName="revoke-keys"
Oct 30 01:29:25.259770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.257012    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29f43b8-6662-418c-94ea-c0be9254013f" containerName="revoke-keys"
Oct 30 01:29:25.259770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.257052    5104 cpu_manager.go:414] "containerMap: removing container" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy"
Oct 30 01:29:25.259770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.257060    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy"
Oct 30 01:29:25.259770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.257281    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" containerName="istio-proxy"
Oct 30 01:29:25.259770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.257304    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" containerName="istio-proxy"
Oct 30 01:29:25.259770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.257324    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29f43b8-6662-418c-94ea-c0be9254013f" containerName="revoke-keys"
Oct 30 01:29:25.263709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.263580    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"]
Oct 30 01:29:25.263915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.263686    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.267379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.267353    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"]
Oct 30 01:29:25.273648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.273626    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"]
Oct 30 01:29:25.273834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.273738    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.278386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.278362    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eaf753\""
Oct 30 01:29:25.278553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.278496    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eaf753-dockercfg-kxvdj\""
Oct 30 01:29:25.298741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.298705    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de81acc9-6f4e-4d26-81ff-50652d037811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.298901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.298855    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.298954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.298910    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5266\" (UniqueName: \"kubernetes.io/projected/de81acc9-6f4e-4d26-81ff-50652d037811-kube-api-access-n5266\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.299132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.299099    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgvf\" (UniqueName: \"kubernetes.io/projected/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-kube-api-access-mlgvf\") pod \"payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.400191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.400099    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.400191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.400157    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5266\" (UniqueName: \"kubernetes.io/projected/de81acc9-6f4e-4d26-81ff-50652d037811-kube-api-access-n5266\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.400409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.400229    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgvf\" (UniqueName: \"kubernetes.io/projected/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-kube-api-access-mlgvf\") pod \"payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.400526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.400413    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de81acc9-6f4e-4d26-81ff-50652d037811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.400976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.400956    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.401227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.401206    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de81acc9-6f4e-4d26-81ff-50652d037811-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.418680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.418646    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5266\" (UniqueName: \"kubernetes.io/projected/de81acc9-6f4e-4d26-81ff-50652d037811-kube-api-access-n5266\") pod \"payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.418845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.418650    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgvf\" (UniqueName: \"kubernetes.io/projected/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-kube-api-access-mlgvf\") pod \"payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.582732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.582691    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:25.593194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.593159    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:25.895842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.895813    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"]
Oct 30 01:29:25.896308 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:25.896276    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde81acc9_6f4e_4d26_81ff_50652d037811.slice/crio-3d67c00b2d5759318ecfd9c8ff2dd4e3d334e1d45f1784d603edd9a7f00cccf5 WatchSource:0}: Error finding container 3d67c00b2d5759318ecfd9c8ff2dd4e3d334e1d45f1784d603edd9a7f00cccf5: Status 404 returned error can't find the container with id 3d67c00b2d5759318ecfd9c8ff2dd4e3d334e1d45f1784d603edd9a7f00cccf5
Oct 30 01:29:25.935320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:25.934585    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"]
Oct 30 01:29:26.581347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.581309    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab5a5f5-36f7-4f21-979f-c013a3e65d2b" path="/var/lib/kubelet/pods/dab5a5f5-36f7-4f21-979f-c013a3e65d2b/volumes"
Oct 30 01:29:26.582034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.582008    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ac2245-ed16-4744-b7c8-cc67c20b47b7" path="/var/lib/kubelet/pods/e4ac2245-ed16-4744-b7c8-cc67c20b47b7/volumes"
Oct 30 01:29:26.804863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.804822    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p" event={"ID":"de81acc9-6f4e-4d26-81ff-50652d037811","Type":"ContainerStarted","Data":"110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f"}
Oct 30 01:29:26.805067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.804889    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p" event={"ID":"de81acc9-6f4e-4d26-81ff-50652d037811","Type":"ContainerStarted","Data":"3d67c00b2d5759318ecfd9c8ff2dd4e3d334e1d45f1784d603edd9a7f00cccf5"}
Oct 30 01:29:26.805067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.804976    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:26.807061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.807016    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9" event={"ID":"6768328a-78ee-4d01-9e6c-df92c1c6a8d4","Type":"ContainerStarted","Data":"1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506"}
Oct 30 01:29:26.807240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.807074    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9" event={"ID":"6768328a-78ee-4d01-9e6c-df92c1c6a8d4","Type":"ContainerStarted","Data":"9aebb30b16e1d785bd133c5bfbe4d3a29bb338bf7b35901cd61245502e6e2aa4"}
Oct 30 01:29:26.807240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.807104    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:26.844307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.844216    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p" podStartSLOduration=1.8441988459999998 podStartE2EDuration="1.844198846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:26.843420666 +0000 UTC m=+2808.391111164" watchObservedRunningTime="2026-10-30 01:29:26.844198846 +0000 UTC m=+2808.391889342"
Oct 30 01:29:26.885369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:26.885310    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9" podStartSLOduration=1.8852947310000001 podStartE2EDuration="1.885294731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:26.880640985 +0000 UTC m=+2808.428331482" watchObservedRunningTime="2026-10-30 01:29:26.885294731 +0000 UTC m=+2808.432985226"
Oct 30 01:29:27.612090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:27.612039    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" podUID="4905cca2-375e-46a5-9845-38ab8171ea14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:29:27.612651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:27.612281    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" podUID="f0b4d577-6ef2-427c-8bd9-527086752e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:29:28.683895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:28.683858    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"]
Oct 30 01:29:28.684394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:28.684123    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy" containerID="cri-o://3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172" gracePeriod=30
Oct 30 01:29:29.377632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.377590    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"]
Oct 30 01:29:29.384417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.384384    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.392954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.392921    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.393144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.392975    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.393347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.393316    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zpv\" (UniqueName: \"kubernetes.io/projected/2530c279-91a4-4f4f-92b2-29f21d4247d4-kube-api-access-82zpv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.404094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.404065    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"]
Oct 30 01:29:29.493703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.493670    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:29.493903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.493776    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:29:29.494418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.494394    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zpv\" (UniqueName: \"kubernetes.io/projected/2530c279-91a4-4f4f-92b2-29f21d4247d4-kube-api-access-82zpv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.494537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.494505    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.494603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.494542    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.495358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.495334    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.497429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.497403    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.514695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.514666    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zpv\" (UniqueName: \"kubernetes.io/projected/2530c279-91a4-4f4f-92b2-29f21d4247d4-kube-api-access-82zpv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:29.706500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:29.705531    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:30.056346 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:30.056312    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2530c279_91a4_4f4f_92b2_29f21d4247d4.slice/crio-85be2c4e4279547537159b356b54768be6bc6758b0c30f4ecc5468f0090c8ef5 WatchSource:0}: Error finding container 85be2c4e4279547537159b356b54768be6bc6758b0c30f4ecc5468f0090c8ef5: Status 404 returned error can't find the container with id 85be2c4e4279547537159b356b54768be6bc6758b0c30f4ecc5468f0090c8ef5
Oct 30 01:29:30.056665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:30.056638    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"]
Oct 30 01:29:31.023241 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:31.023196    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc" event={"ID":"2530c279-91a4-4f4f-92b2-29f21d4247d4","Type":"ContainerStarted","Data":"04826a8dbda1c905c25ce8cb01dbbfac69d018f651b1d88213315544535aa509"}
Oct 30 01:29:31.023734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:31.023268    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc" event={"ID":"2530c279-91a4-4f4f-92b2-29f21d4247d4","Type":"ContainerStarted","Data":"85be2c4e4279547537159b356b54768be6bc6758b0c30f4ecc5468f0090c8ef5"}
Oct 30 01:29:31.065545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:31.065488    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc" podStartSLOduration=2.065474087 podStartE2EDuration="2.065474087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:31.06152379 +0000 UTC m=+2812.609214308" watchObservedRunningTime="2026-10-30 01:29:31.065474087 +0000 UTC m=+2812.613164577"
Oct 30 01:29:32.033922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:32.033885    5104 generic.go:356] "Generic (PLEG): container finished" podID="2530c279-91a4-4f4f-92b2-29f21d4247d4" containerID="04826a8dbda1c905c25ce8cb01dbbfac69d018f651b1d88213315544535aa509" exitCode=0
Oct 30 01:29:32.034428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:32.033973    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc" event={"ID":"2530c279-91a4-4f4f-92b2-29f21d4247d4","Type":"ContainerDied","Data":"04826a8dbda1c905c25ce8cb01dbbfac69d018f651b1d88213315544535aa509"}
Oct 30 01:29:33.281227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.281199    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:33.412829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.412739    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-openshift-ca\") pod \"2530c279-91a4-4f4f-92b2-29f21d4247d4\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") "
Oct 30 01:29:33.412829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.412799    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2530c279-91a4-4f4f-92b2-29f21d4247d4-kube-api-access-82zpv\" (UniqueName: \"kubernetes.io/projected/2530c279-91a4-4f4f-92b2-29f21d4247d4-kube-api-access-82zpv\") pod \"2530c279-91a4-4f4f-92b2-29f21d4247d4\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") "
Oct 30 01:29:33.413123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.413096    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-openshift-ca" pod "2530c279-91a4-4f4f-92b2-29f21d4247d4" (UID: "2530c279-91a4-4f4f-92b2-29f21d4247d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:33.413195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.413176    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-xks-ca\") pod \"2530c279-91a4-4f4f-92b2-29f21d4247d4\" (UID: \"2530c279-91a4-4f4f-92b2-29f21d4247d4\") "
Oct 30 01:29:33.413762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.413738    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:33.415590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.415565    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-xks-ca" pod "2530c279-91a4-4f4f-92b2-29f21d4247d4" (UID: "2530c279-91a4-4f4f-92b2-29f21d4247d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:33.415732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.415582    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2530c279-91a4-4f4f-92b2-29f21d4247d4-kube-api-access-82zpv" pod "2530c279-91a4-4f4f-92b2-29f21d4247d4" (UID: "2530c279-91a4-4f4f-92b2-29f21d4247d4"). InnerVolumeSpecName "kube-api-access-82zpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:33.514946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.514908    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2530c279-91a4-4f4f-92b2-29f21d4247d4-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:33.514946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.514944    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82zpv\" (UniqueName: \"kubernetes.io/projected/2530c279-91a4-4f4f-92b2-29f21d4247d4-kube-api-access-82zpv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:33.966848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:33.966820    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:29:34.023326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023291    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-token\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-token\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.023543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023529    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-credential-socket\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.023676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023651    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-socket\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.023785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023768    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-certs\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.023852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023822    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d185da3d-11d2-401a-9c03-73f557d4d7f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d185da3d-11d2-401a-9c03-73f557d4d7f2-istiod-ca-cert\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.023917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023859    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-envoy\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.023917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023904    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-credential-socket" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:34.024039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.023981    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-data\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.024039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.024014    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-kube-api-access-pzfjt\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-kube-api-access-pzfjt\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.024153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.024076    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-podinfo\") pod \"d185da3d-11d2-401a-9c03-73f557d4d7f2\" (UID: \"d185da3d-11d2-401a-9c03-73f557d4d7f2\") "
Oct 30 01:29:34.024218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.024194    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-certs" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:34.024594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.024269    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-socket" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:34.024594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.024504    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-data" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:34.024828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.024805    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d185da3d-11d2-401a-9c03-73f557d4d7f2-istiod-ca-cert" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:34.026444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.026415    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.026578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.026444    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d185da3d-11d2-401a-9c03-73f557d4d7f2-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.026578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.026474    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.026578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.026487    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.026578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.026502    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.026962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.026941    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-token" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:34.027071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.027040    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-envoy" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:34.028747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.028712    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-podinfo" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:34.029004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.028977    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-kube-api-access-pzfjt" pod "d185da3d-11d2-401a-9c03-73f557d4d7f2" (UID: "d185da3d-11d2-401a-9c03-73f557d4d7f2"). InnerVolumeSpecName "kube-api-access-pzfjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:34.055226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.055192    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"
Oct 30 01:29:34.055423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.055222    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc" event={"ID":"2530c279-91a4-4f4f-92b2-29f21d4247d4","Type":"ContainerDied","Data":"85be2c4e4279547537159b356b54768be6bc6758b0c30f4ecc5468f0090c8ef5"}
Oct 30 01:29:34.055423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.055290    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85be2c4e4279547537159b356b54768be6bc6758b0c30f4ecc5468f0090c8ef5"
Oct 30 01:29:34.058085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.058052    5104 generic.go:356] "Generic (PLEG): container finished" podID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerID="3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172" exitCode=0
Oct 30 01:29:34.058258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.058112    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"
Oct 30 01:29:34.058258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.058112    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" event={"ID":"d185da3d-11d2-401a-9c03-73f557d4d7f2","Type":"ContainerDied","Data":"3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172"}
Oct 30 01:29:34.058258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.058247    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj" event={"ID":"d185da3d-11d2-401a-9c03-73f557d4d7f2","Type":"ContainerDied","Data":"7abb7fd4312c8e7199cbcf54ca43ce2f3a382310657e1309b02e8a9dcbe70f3a"}
Oct 30 01:29:34.058416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.058286    5104 scope.go:122] "RemoveContainer" containerID="3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172"
Oct 30 01:29:34.074981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.074956    5104 scope.go:122] "RemoveContainer" containerID="fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1"
Oct 30 01:29:34.089961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.089939    5104 scope.go:122] "RemoveContainer" containerID="3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172"
Oct 30 01:29:34.090305 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:34.090282    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172\": container with ID starting with 3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172 not found: ID does not exist" containerID="3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172"
Oct 30 01:29:34.090398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.090315    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172"} err="failed to get container status \"3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172\": rpc error: code = NotFound desc = could not find container \"3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172\": container with ID starting with 3736b304b61237ed31009051d4de9c02d5258510929442cc93fe8a95b0083172 not found: ID does not exist"
Oct 30 01:29:34.090398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.090332    5104 scope.go:122] "RemoveContainer" containerID="fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1"
Oct 30 01:29:34.090638 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:34.090621    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1\": container with ID starting with fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1 not found: ID does not exist" containerID="fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1"
Oct 30 01:29:34.090708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.090644    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1"} err="failed to get container status \"fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1\": rpc error: code = NotFound desc = could not find container \"fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1\": container with ID starting with fbc338dc39888e72296051e2a79cd75e1afc3be58ceaf235e2e24672b73277a1 not found: ID does not exist"
Oct 30 01:29:34.126711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.126676    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"]
Oct 30 01:29:34.127992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.127958    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.127992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.127993    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzfjt\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-kube-api-access-pzfjt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.128202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.128007    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.128202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.128020    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d185da3d-11d2-401a-9c03-73f557d4d7f2-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:34.131364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.131327    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-gk5cj"]
Oct 30 01:29:34.580829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:34.580751    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" path="/var/lib/kubelet/pods/d185da3d-11d2-401a-9c03-73f557d4d7f2/volumes"
Oct 30 01:29:37.199640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.199610    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"]
Oct 30 01:29:37.305524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.287646    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-999ff9-85cfk"]
Oct 30 01:29:37.606567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.606466    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"]
Oct 30 01:29:37.610687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.610653    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-add1bb-zptrp"]
Oct 30 01:29:37.612619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.612584    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" podUID="4905cca2-375e-46a5-9845-38ab8171ea14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:29:37.612784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.612619    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" podUID="f0b4d577-6ef2-427c-8bd9-527086752e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:29:37.864946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.864853    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:29:37.864946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:37.864927    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:29:38.204864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:38.204823    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"]
Oct 30 01:29:38.209055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:38.209021    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"]
Oct 30 01:29:38.608842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:38.608234    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ffecf2-3ee2-4217-9072-e1cf151ea0e0" path="/var/lib/kubelet/pods/21ffecf2-3ee2-4217-9072-e1cf151ea0e0/volumes"
Oct 30 01:29:38.609843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:38.609815    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c92fb3-bef5-49c8-bb28-7e9b4edc3b79" path="/var/lib/kubelet/pods/94c92fb3-bef5-49c8-bb28-7e9b4edc3b79/volumes"
Oct 30 01:29:39.232509 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:39.231841    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:29:39.295487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:39.273063    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main" containerID="cri-o://d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6" gracePeriod=45
Oct 30 01:29:39.309146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:39.309100    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" containerID="cri-o://36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e" gracePeriod=45
Oct 30 01:29:39.507436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:39.505642    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": dial tcp 10.132.0.69:8000: connect: connection refused"
Oct 30 01:29:39.928567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:39.928534    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:29:39.955618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:39.955432    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:29:40.027519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.027473    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-dshm\") pod \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") "
Oct 30 01:29:40.027778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.027576    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tls-certs\") pod \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") "
Oct 30 01:29:40.027778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.027647    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-model-cache\") pod \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") "
Oct 30 01:29:40.027778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.027697    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tls-certs\" (UniqueName: \"kubernetes.io/secret/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tls-certs\") pod \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") "
Oct 30 01:29:40.027778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.027757    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-kube-api-access-9c5b2\" (UniqueName: \"kubernetes.io/projected/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-kube-api-access-9c5b2\") pod \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") "
Oct 30 01:29:40.028015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.027931    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-model-cache\") pod \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") "
Oct 30 01:29:40.028199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.028165    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-model-cache" pod "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" (UID: "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.029182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.028497    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tmp-dir\") pod \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") "
Oct 30 01:29:40.029182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.028565    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tmp-dir\") pod \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") "
Oct 30 01:29:40.029182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.028600    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-home\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-home\") pod \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") "
Oct 30 01:29:40.029182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.028636    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-home\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-home\") pod \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") "
Oct 30 01:29:40.029182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.028702    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69a955d-b56b-4dcd-943a-c1af4d8eb363-kube-api-access-7ckjr\" (UniqueName: \"kubernetes.io/projected/e69a955d-b56b-4dcd-943a-c1af4d8eb363-kube-api-access-7ckjr\") pod \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\" (UID: \"e69a955d-b56b-4dcd-943a-c1af4d8eb363\") "
Oct 30 01:29:40.029182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.028759    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-dshm\") pod \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\" (UID: \"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f\") "
Oct 30 01:29:40.029631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.029202    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-model-cache" pod "e69a955d-b56b-4dcd-943a-c1af4d8eb363" (UID: "e69a955d-b56b-4dcd-943a-c1af4d8eb363"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.029631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.029391    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-home" pod "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" (UID: "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.029631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.029591    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tmp-dir" pod "e69a955d-b56b-4dcd-943a-c1af4d8eb363" (UID: "e69a955d-b56b-4dcd-943a-c1af4d8eb363"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.030428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.030396    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-home" pod "e69a955d-b56b-4dcd-943a-c1af4d8eb363" (UID: "e69a955d-b56b-4dcd-943a-c1af4d8eb363"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.031401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.031382    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tmp-dir" pod "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" (UID: "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.032420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.032389    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-dshm" pod "e69a955d-b56b-4dcd-943a-c1af4d8eb363" (UID: "e69a955d-b56b-4dcd-943a-c1af4d8eb363"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.032963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.032942    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.033080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033068    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.033150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033141    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.033204 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033197    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.033263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033256    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.033319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033313    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.033371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033364    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69a955d-b56b-4dcd-943a-c1af4d8eb363-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.033508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033481    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-kube-api-access-9c5b2" pod "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" (UID: "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f"). InnerVolumeSpecName "kube-api-access-9c5b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:40.033805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.033786    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-dshm" pod "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" (UID: "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:40.035428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.035401    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69a955d-b56b-4dcd-943a-c1af4d8eb363-kube-api-access-7ckjr" pod "e69a955d-b56b-4dcd-943a-c1af4d8eb363" (UID: "e69a955d-b56b-4dcd-943a-c1af4d8eb363"). InnerVolumeSpecName "kube-api-access-7ckjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:40.036248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.036223    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tls-certs" pod "e69a955d-b56b-4dcd-943a-c1af4d8eb363" (UID: "e69a955d-b56b-4dcd-943a-c1af4d8eb363"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:40.036344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.036327    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tls-certs" pod "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" (UID: "8bc83b57-7ac2-4457-8b3c-0c844b77bb8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:40.135335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.135292    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.135647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.135627    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69a955d-b56b-4dcd-943a-c1af4d8eb363-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.139356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.139325    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c5b2\" (UniqueName: \"kubernetes.io/projected/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-kube-api-access-9c5b2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.139584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.139564    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ckjr\" (UniqueName: \"kubernetes.io/projected/e69a955d-b56b-4dcd-943a-c1af4d8eb363-kube-api-access-7ckjr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.139726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.139708    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.217693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.217662    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"]
Oct 30 01:29:40.220860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.219999    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:29:40.238815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.238724    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:29:40.344134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.344079    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4905cca2-375e-46a5-9845-38ab8171ea14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905cca2-375e-46a5-9845-38ab8171ea14-plugins-config-volume\") pod \"4905cca2-375e-46a5-9845-38ab8171ea14\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") "
Oct 30 01:29:40.344301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.344142    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4905cca2-375e-46a5-9845-38ab8171ea14-kube-api-access-64rd2\" (UniqueName: \"kubernetes.io/projected/4905cca2-375e-46a5-9845-38ab8171ea14-kube-api-access-64rd2\") pod \"4905cca2-375e-46a5-9845-38ab8171ea14\" (UID: \"4905cca2-375e-46a5-9845-38ab8171ea14\") "
Oct 30 01:29:40.344301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.344225    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b4d577-6ef2-427c-8bd9-527086752e20-kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/f0b4d577-6ef2-427c-8bd9-527086752e20-kube-api-access-xz9bj\") pod \"f0b4d577-6ef2-427c-8bd9-527086752e20\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") "
Oct 30 01:29:40.344798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.344509    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b4d577-6ef2-427c-8bd9-527086752e20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b4d577-6ef2-427c-8bd9-527086752e20-plugins-config-volume\") pod \"f0b4d577-6ef2-427c-8bd9-527086752e20\" (UID: \"f0b4d577-6ef2-427c-8bd9-527086752e20\") "
Oct 30 01:29:40.345544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.345517    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4905cca2-375e-46a5-9845-38ab8171ea14-plugins-config-volume" pod "4905cca2-375e-46a5-9845-38ab8171ea14" (UID: "4905cca2-375e-46a5-9845-38ab8171ea14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:40.345701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.345564    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b4d577-6ef2-427c-8bd9-527086752e20-plugins-config-volume" pod "f0b4d577-6ef2-427c-8bd9-527086752e20" (UID: "f0b4d577-6ef2-427c-8bd9-527086752e20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:40.348308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.348227    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4905cca2-375e-46a5-9845-38ab8171ea14-kube-api-access-64rd2" pod "4905cca2-375e-46a5-9845-38ab8171ea14" (UID: "4905cca2-375e-46a5-9845-38ab8171ea14"). InnerVolumeSpecName "kube-api-access-64rd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:40.348894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.348832    5104 generic.go:356] "Generic (PLEG): container finished" podID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerID="d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6" exitCode=0
Oct 30 01:29:40.349079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.349061    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"
Oct 30 01:29:40.351620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.349065    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" event={"ID":"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f","Type":"ContainerDied","Data":"d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6"}
Oct 30 01:29:40.351731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.351653    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth" event={"ID":"8bc83b57-7ac2-4457-8b3c-0c844b77bb8f","Type":"ContainerDied","Data":"d031f708e312cb34f2333e65d78ae898621d1758a971de0318af47cc6f2828e0"}
Oct 30 01:29:40.351731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.351691    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" event={"ID":"f0b4d577-6ef2-427c-8bd9-527086752e20","Type":"ContainerDied","Data":"2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56"}
Oct 30 01:29:40.351847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.351516    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"
Oct 30 01:29:40.351847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.350134    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b4d577-6ef2-427c-8bd9-527086752e20-kube-api-access-xz9bj" pod "f0b4d577-6ef2-427c-8bd9-527086752e20" (UID: "f0b4d577-6ef2-427c-8bd9-527086752e20"). InnerVolumeSpecName "kube-api-access-xz9bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:40.351973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.351856    5104 scope.go:122] "RemoveContainer" containerID="d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6"
Oct 30 01:29:40.352055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.351406    5104 generic.go:356] "Generic (PLEG): container finished" podID="f0b4d577-6ef2-427c-8bd9-527086752e20" containerID="2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56" exitCode=0
Oct 30 01:29:40.352227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.352206    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq" event={"ID":"f0b4d577-6ef2-427c-8bd9-527086752e20","Type":"ContainerDied","Data":"c272bf5c29d4084f2a4974ccf7d4acfd1680c5b202d9e89c8d75325325df20fb"}
Oct 30 01:29:40.356728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.354773    5104 generic.go:356] "Generic (PLEG): container finished" podID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerID="36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e" exitCode=0
Oct 30 01:29:40.356728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.354893    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" event={"ID":"e69a955d-b56b-4dcd-943a-c1af4d8eb363","Type":"ContainerDied","Data":"36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e"}
Oct 30 01:29:40.356728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.354919    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc" event={"ID":"e69a955d-b56b-4dcd-943a-c1af4d8eb363","Type":"ContainerDied","Data":"b188667233dfe138234983122f0d72a9b2b9c368b0a57a4a7ff0bc71e825daf1"}
Oct 30 01:29:40.356728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.354989    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"
Oct 30 01:29:40.364154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.364116    5104 generic.go:356] "Generic (PLEG): container finished" podID="4905cca2-375e-46a5-9845-38ab8171ea14" containerID="b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772" exitCode=0
Oct 30 01:29:40.364295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.364231    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"
Oct 30 01:29:40.364295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.364271    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" event={"ID":"4905cca2-375e-46a5-9845-38ab8171ea14","Type":"ContainerDied","Data":"b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772"}
Oct 30 01:29:40.364423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.364346    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn" event={"ID":"4905cca2-375e-46a5-9845-38ab8171ea14","Type":"ContainerDied","Data":"76b3569bb1594eec259acd00678ce5cbc1e9ee12afbace0703475e078bb66f56"}
Oct 30 01:29:40.455335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.455291    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b4d577-6ef2-427c-8bd9-527086752e20-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.455335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.455331    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4905cca2-375e-46a5-9845-38ab8171ea14-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.455624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.455347    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rd2\" (UniqueName: \"kubernetes.io/projected/4905cca2-375e-46a5-9845-38ab8171ea14-kube-api-access-64rd2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.455624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.455362    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9bj\" (UniqueName: \"kubernetes.io/projected/f0b4d577-6ef2-427c-8bd9-527086752e20-kube-api-access-xz9bj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:40.516270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.515890    5104 scope.go:122] "RemoveContainer" containerID="d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6"
Oct 30 01:29:40.517406 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:40.517261    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6\": container with ID starting with d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6 not found: ID does not exist" containerID="d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6"
Oct 30 01:29:40.517406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.517299    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6"} err="failed to get container status \"d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6\": rpc error: code = NotFound desc = could not find container \"d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6\": container with ID starting with d37e39b35b1f05a9c48e087c7a10ae3bdfa41c5fc232b37af5c82ea8a35ff7b6 not found: ID does not exist"
Oct 30 01:29:40.517406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.517326    5104 scope.go:122] "RemoveContainer" containerID="2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56"
Oct 30 01:29:40.538807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.538396    5104 scope.go:122] "RemoveContainer" containerID="2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56"
Oct 30 01:29:40.539638 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:40.539275    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56\": container with ID starting with 2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56 not found: ID does not exist" containerID="2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56"
Oct 30 01:29:40.539638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.539315    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56"} err="failed to get container status \"2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56\": rpc error: code = NotFound desc = could not find container \"2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56\": container with ID starting with 2ecac6b9e82fb1c9a17ab6fa080f6a3e2f247091c9e0786add108e46372eab56 not found: ID does not exist"
Oct 30 01:29:40.539638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.539341    5104 scope.go:122] "RemoveContainer" containerID="36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e"
Oct 30 01:29:40.540238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.540189    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"]
Oct 30 01:29:40.544587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.544564    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-64f472-588bdcb8c6-rhgnq"]
Oct 30 01:29:40.554412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.554390    5104 scope.go:122] "RemoveContainer" containerID="36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e"
Oct 30 01:29:40.554792 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:40.554751    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e\": container with ID starting with 36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e not found: ID does not exist" containerID="36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e"
Oct 30 01:29:40.554792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.554785    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e"} err="failed to get container status \"36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e\": rpc error: code = NotFound desc = could not find container \"36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e\": container with ID starting with 36f3744f1538e2aaf24f27fd973472ea5dba39ebecb3fce764fc973de1224a3e not found: ID does not exist"
Oct 30 01:29:40.555014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.554805    5104 scope.go:122] "RemoveContainer" containerID="b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772"
Oct 30 01:29:40.622337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.622315    5104 scope.go:122] "RemoveContainer" containerID="b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772"
Oct 30 01:29:40.622970 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:40.622935    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772\": container with ID starting with b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772 not found: ID does not exist" containerID="b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772"
Oct 30 01:29:40.623198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.623164    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772"} err="failed to get container status \"b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772\": rpc error: code = NotFound desc = could not find container \"b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772\": container with ID starting with b881cc242169bde8af88d8bc5d520114d1bae61be1712a0b53fd313750440772 not found: ID does not exist"
Oct 30 01:29:40.636251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.636223    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b4d577-6ef2-427c-8bd9-527086752e20" path="/var/lib/kubelet/pods/f0b4d577-6ef2-427c-8bd9-527086752e20/volumes"
Oct 30 01:29:40.636610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.636595    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"]
Oct 30 01:29:40.636675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.636625    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/premium-w2-fd3281-kserve-7ffcc4799b-58zkc"]
Oct 30 01:29:40.653281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.653254    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"]
Oct 30 01:29:40.657928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.657900    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-64f472-77bc5f89cf-pfpfn"]
Oct 30 01:29:40.711862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.711824    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"]
Oct 30 01:29:40.718056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:40.718020    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-w2-fd3281-kserve-7b4f5d499-8kzth"]
Oct 30 01:29:42.604905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:42.604869    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4905cca2-375e-46a5-9845-38ab8171ea14" path="/var/lib/kubelet/pods/4905cca2-375e-46a5-9845-38ab8171ea14/volumes"
Oct 30 01:29:42.605420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:42.605394    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" path="/var/lib/kubelet/pods/8bc83b57-7ac2-4457-8b3c-0c844b77bb8f/volumes"
Oct 30 01:29:42.605975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:42.605954    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" path="/var/lib/kubelet/pods/e69a955d-b56b-4dcd-943a-c1af4d8eb363/volumes"
Oct 30 01:29:43.207089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.205544    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p" podUID="e5e39f16-1cf7-4311-9d97-4cd699f8d308" containerName="payload-processing" containerID="cri-o://e2041b7e740e7d8dcc800e586c653c354ba6f2c4d8b4d48b0893b350d4d8891d" gracePeriod=25
Oct 30 01:29:43.211637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.211544    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd" podUID="42544934-4a64-4f8b-8c7d-5b13dacbe54a" containerName="payload-pre-processing" containerID="cri-o://c2c266ad76aed5b961d8403614c074870266ada76dc99c92594aeb55caca1396" gracePeriod=25
Oct 30 01:29:43.363227 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.363171    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"]
Oct 30 01:29:43.364436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.364411    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main"
Oct 30 01:29:43.364644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.364628    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main"
Oct 30 01:29:43.364767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.364755    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy"
Oct 30 01:29:43.364846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.364835    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy"
Oct 30 01:29:43.364964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.364954    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f0b4d577-6ef2-427c-8bd9-527086752e20" containerName="payload-processing"
Oct 30 01:29:43.365060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365049    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b4d577-6ef2-427c-8bd9-527086752e20" containerName="payload-processing"
Oct 30 01:29:43.365146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365136    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy"
Oct 30 01:29:43.365225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365214    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy"
Oct 30 01:29:43.365326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365316    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2530c279-91a4-4f4f-92b2-29f21d4247d4" containerName="revoke-keys"
Oct 30 01:29:43.365410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365399    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2530c279-91a4-4f4f-92b2-29f21d4247d4" containerName="revoke-keys"
Oct 30 01:29:43.365534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365522    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4905cca2-375e-46a5-9845-38ab8171ea14" containerName="payload-pre-processing"
Oct 30 01:29:43.365627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365615    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4905cca2-375e-46a5-9845-38ab8171ea14" containerName="payload-pre-processing"
Oct 30 01:29:43.365719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365710    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main"
Oct 30 01:29:43.365792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.365783    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main"
Oct 30 01:29:43.366121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.366104    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4905cca2-375e-46a5-9845-38ab8171ea14" containerName="payload-pre-processing"
Oct 30 01:29:43.366287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.366273    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc83b57-7ac2-4457-8b3c-0c844b77bb8f" containerName="main"
Oct 30 01:29:43.366383 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.366372    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69a955d-b56b-4dcd-943a-c1af4d8eb363" containerName="main"
Oct 30 01:29:43.366492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.366480    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy"
Oct 30 01:29:43.366605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.366593    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b4d577-6ef2-427c-8bd9-527086752e20" containerName="payload-processing"
Oct 30 01:29:43.366689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.366674    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d185da3d-11d2-401a-9c03-73f557d4d7f2" containerName="istio-proxy"
Oct 30 01:29:43.366781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.366769    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2530c279-91a4-4f4f-92b2-29f21d4247d4" containerName="revoke-keys"
Oct 30 01:29:43.372166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.371438    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.388441 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.388406    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"]
Oct 30 01:29:43.391865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.391623    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:29:43.391865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.391704    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:43.438741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.438699    5104 generic.go:356] "Generic (PLEG): container finished" podID="e5e39f16-1cf7-4311-9d97-4cd699f8d308" containerID="e2041b7e740e7d8dcc800e586c653c354ba6f2c4d8b4d48b0893b350d4d8891d" exitCode=0
Oct 30 01:29:43.438903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.438792    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p" event={"ID":"e5e39f16-1cf7-4311-9d97-4cd699f8d308","Type":"ContainerDied","Data":"e2041b7e740e7d8dcc800e586c653c354ba6f2c4d8b4d48b0893b350d4d8891d"}
Oct 30 01:29:43.441190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.441159    5104 generic.go:356] "Generic (PLEG): container finished" podID="42544934-4a64-4f8b-8c7d-5b13dacbe54a" containerID="c2c266ad76aed5b961d8403614c074870266ada76dc99c92594aeb55caca1396" exitCode=0
Oct 30 01:29:43.441314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.441207    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd" event={"ID":"42544934-4a64-4f8b-8c7d-5b13dacbe54a","Type":"ContainerDied","Data":"c2c266ad76aed5b961d8403614c074870266ada76dc99c92594aeb55caca1396"}
Oct 30 01:29:43.543164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.543130    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.543319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.543176    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6pv\" (UniqueName: \"kubernetes.io/projected/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-kube-api-access-hs6pv\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.543319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.543210    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.629992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.629596    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:29:43.644679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.644645    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.645003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.644879    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6pv\" (UniqueName: \"kubernetes.io/projected/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-kube-api-access-hs6pv\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.645003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.644965    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.645655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.645627    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.647371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.647350    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.665906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.665882    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6pv\" (UniqueName: \"kubernetes.io/projected/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-kube-api-access-hs6pv\") pod \"maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.666706 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.666687    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:29:43.721095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.721041    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:43.747076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.747012    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e39f16-1cf7-4311-9d97-4cd699f8d308-kube-api-access-mvnhd\" (UniqueName: \"kubernetes.io/projected/e5e39f16-1cf7-4311-9d97-4cd699f8d308-kube-api-access-mvnhd\") pod \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") "
Oct 30 01:29:43.747223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.747119    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42544934-4a64-4f8b-8c7d-5b13dacbe54a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42544934-4a64-4f8b-8c7d-5b13dacbe54a-plugins-config-volume\") pod \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") "
Oct 30 01:29:43.747223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.747147    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e39f16-1cf7-4311-9d97-4cd699f8d308-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e39f16-1cf7-4311-9d97-4cd699f8d308-plugins-config-volume\") pod \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\" (UID: \"e5e39f16-1cf7-4311-9d97-4cd699f8d308\") "
Oct 30 01:29:43.747223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.747221    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42544934-4a64-4f8b-8c7d-5b13dacbe54a-kube-api-access-mfp2k\" (UniqueName: \"kubernetes.io/projected/42544934-4a64-4f8b-8c7d-5b13dacbe54a-kube-api-access-mfp2k\") pod \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\" (UID: \"42544934-4a64-4f8b-8c7d-5b13dacbe54a\") "
Oct 30 01:29:43.748781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.748752    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42544934-4a64-4f8b-8c7d-5b13dacbe54a-plugins-config-volume" pod "42544934-4a64-4f8b-8c7d-5b13dacbe54a" (UID: "42544934-4a64-4f8b-8c7d-5b13dacbe54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:43.749213 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.749179    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e39f16-1cf7-4311-9d97-4cd699f8d308-plugins-config-volume" pod "e5e39f16-1cf7-4311-9d97-4cd699f8d308" (UID: "e5e39f16-1cf7-4311-9d97-4cd699f8d308"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:43.753415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.753373    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42544934-4a64-4f8b-8c7d-5b13dacbe54a-kube-api-access-mfp2k" pod "42544934-4a64-4f8b-8c7d-5b13dacbe54a" (UID: "42544934-4a64-4f8b-8c7d-5b13dacbe54a"). InnerVolumeSpecName "kube-api-access-mfp2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:43.757476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.757434    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e39f16-1cf7-4311-9d97-4cd699f8d308-kube-api-access-mvnhd" pod "e5e39f16-1cf7-4311-9d97-4cd699f8d308" (UID: "e5e39f16-1cf7-4311-9d97-4cd699f8d308"). InnerVolumeSpecName "kube-api-access-mvnhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:43.849156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.849117    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvnhd\" (UniqueName: \"kubernetes.io/projected/e5e39f16-1cf7-4311-9d97-4cd699f8d308-kube-api-access-mvnhd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:43.849156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.849157    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42544934-4a64-4f8b-8c7d-5b13dacbe54a-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:43.849684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.849172    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e39f16-1cf7-4311-9d97-4cd699f8d308-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:43.849684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:43.849186    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfp2k\" (UniqueName: \"kubernetes.io/projected/42544934-4a64-4f8b-8c7d-5b13dacbe54a-kube-api-access-mfp2k\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:44.045569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.045515    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"]
Oct 30 01:29:44.047775 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:44.047730    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea31cb1_fd82_411f_ae4e_5a0b1f0fc1eb.slice/crio-bf5efbb4102d0a9a02d9dacd0e00b9f860ce440b87ce0420257ac48d08026977 WatchSource:0}: Error finding container bf5efbb4102d0a9a02d9dacd0e00b9f860ce440b87ce0420257ac48d08026977: Status 404 returned error can't find the container with id bf5efbb4102d0a9a02d9dacd0e00b9f860ce440b87ce0420257ac48d08026977
Oct 30 01:29:44.553425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.544496    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"
Oct 30 01:29:44.553425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.553391    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p" event={"ID":"e5e39f16-1cf7-4311-9d97-4cd699f8d308","Type":"ContainerDied","Data":"5164de178b7a5f4e1a9678964e38a2dbe3ed9e1df390c61e176b1a06777d93d4"}
Oct 30 01:29:44.553691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.553506    5104 scope.go:122] "RemoveContainer" containerID="e2041b7e740e7d8dcc800e586c653c354ba6f2c4d8b4d48b0893b350d4d8891d"
Oct 30 01:29:44.672486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.671002    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"
Oct 30 01:29:44.718794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.717921    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd" event={"ID":"42544934-4a64-4f8b-8c7d-5b13dacbe54a","Type":"ContainerDied","Data":"3a69fbfc0dd822349e273444acce2418a1c6f5e81cf36402b4614b4bc81c58ae"}
Oct 30 01:29:44.754892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.752341    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb" event={"ID":"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb","Type":"ContainerStarted","Data":"bf5efbb4102d0a9a02d9dacd0e00b9f860ce440b87ce0420257ac48d08026977"}
Oct 30 01:29:44.774805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.774345    5104 scope.go:122] "RemoveContainer" containerID="c2c266ad76aed5b961d8403614c074870266ada76dc99c92594aeb55caca1396"
Oct 30 01:29:44.812988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.812938    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"]
Oct 30 01:29:44.824847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.824633    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-55ff675f5d-zjh5p"]
Oct 30 01:29:44.914555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.911344    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"]
Oct 30 01:29:44.917452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.917412    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-bb8bfbb7f-fcnrd"]
Oct 30 01:29:44.992269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.992232    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"]
Oct 30 01:29:44.993226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:44.992953    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy" containerID="cri-o://6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3" gracePeriod=30
Oct 30 01:29:45.218407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.218332    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerName="payload-pre-processing" containerID="cri-o://dd393ab5ad3833c43e68ca0acd607b8c40768b391c464436cdde00e64441b09e" gracePeriod=25
Oct 30 01:29:45.314545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.303769    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"]
Oct 30 01:29:45.314545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.305064    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e5e39f16-1cf7-4311-9d97-4cd699f8d308" containerName="payload-processing"
Oct 30 01:29:45.314545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.305092    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e39f16-1cf7-4311-9d97-4cd699f8d308" containerName="payload-processing"
Oct 30 01:29:45.314545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.305150    5104 cpu_manager.go:414] "containerMap: removing container" podUID="42544934-4a64-4f8b-8c7d-5b13dacbe54a" containerName="payload-pre-processing"
Oct 30 01:29:45.314545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.305159    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42544934-4a64-4f8b-8c7d-5b13dacbe54a" containerName="payload-pre-processing"
Oct 30 01:29:45.314545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.305366    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e39f16-1cf7-4311-9d97-4cd699f8d308" containerName="payload-processing"
Oct 30 01:29:45.314545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.305388    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="42544934-4a64-4f8b-8c7d-5b13dacbe54a" containerName="payload-pre-processing"
Oct 30 01:29:45.339479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.335640    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"]
Oct 30 01:29:45.339479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.335780    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.487388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.487243    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vcs\" (UniqueName: \"kubernetes.io/projected/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-kube-api-access-82vcs\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.487691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.487656    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.487771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.487746    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.590067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.590020    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vcs\" (UniqueName: \"kubernetes.io/projected/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-kube-api-access-82vcs\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.590336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.590315    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.590433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.590382    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.591699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.591661    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.594524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.594484    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.609979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.609942    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vcs\" (UniqueName: \"kubernetes.io/projected/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-kube-api-access-82vcs\") pod \"maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.662747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.662694    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:45.674238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.674197    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"]
Oct 30 01:29:45.686216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.686144    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.692404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.691475    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.692404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.691679    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2nmx\" (UniqueName: \"kubernetes.io/projected/f451ba8b-697a-4bea-88bf-b63e4e1c9690-kube-api-access-m2nmx\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.692404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.691732    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.707784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.705829    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"]
Oct 30 01:29:45.793331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.793273    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2nmx\" (UniqueName: \"kubernetes.io/projected/f451ba8b-697a-4bea-88bf-b63e4e1c9690-kube-api-access-m2nmx\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.804322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.803656    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.804322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.803818    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.805660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.805291    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.814086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.814057    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:45.822196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.822174    5104 generic.go:356] "Generic (PLEG): container finished" podID="4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" containerID="3cdcf68463b329b239df88847702a702539a0af9e947183cf94f680d60e7da34" exitCode=0
Oct 30 01:29:45.822627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.822604    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb" event={"ID":"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb","Type":"ContainerDied","Data":"3cdcf68463b329b239df88847702a702539a0af9e947183cf94f680d60e7da34"}
Oct 30 01:29:45.824690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:45.824672    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2nmx\" (UniqueName: \"kubernetes.io/projected/f451ba8b-697a-4bea-88bf-b63e4e1c9690-kube-api-access-m2nmx\") pod \"maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:46.019885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.019850    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"]
Oct 30 01:29:46.023014 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:46.022979    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c0f7a1_a2a0_4b6c_a0ba_7cc12f88ca33.slice/crio-8796ee10d85d8698bfe30b4701cad216ee37ad6425c7e6ae3eee906147d3a9b2 WatchSource:0}: Error finding container 8796ee10d85d8698bfe30b4701cad216ee37ad6425c7e6ae3eee906147d3a9b2: Status 404 returned error can't find the container with id 8796ee10d85d8698bfe30b4701cad216ee37ad6425c7e6ae3eee906147d3a9b2
Oct 30 01:29:46.078549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.078506    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:46.395199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.395164    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"]
Oct 30 01:29:46.395771 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:46.395741    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf451ba8b_697a_4bea_88bf_b63e4e1c9690.slice/crio-65ef6fbe3fd610094ccb615d3ff8aa4b5f7159df0b738515e053ebd2f3395a30 WatchSource:0}: Error finding container 65ef6fbe3fd610094ccb615d3ff8aa4b5f7159df0b738515e053ebd2f3395a30: Status 404 returned error can't find the container with id 65ef6fbe3fd610094ccb615d3ff8aa4b5f7159df0b738515e053ebd2f3395a30
Oct 30 01:29:46.583524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.583326    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42544934-4a64-4f8b-8c7d-5b13dacbe54a" path="/var/lib/kubelet/pods/42544934-4a64-4f8b-8c7d-5b13dacbe54a/volumes"
Oct 30 01:29:46.584039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.584018    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e39f16-1cf7-4311-9d97-4cd699f8d308" path="/var/lib/kubelet/pods/e5e39f16-1cf7-4311-9d97-4cd699f8d308/volumes"
Oct 30 01:29:46.837070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.837026    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx" event={"ID":"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33","Type":"ContainerStarted","Data":"06fc7cb739ef626c6afadfa5493a0afad35bb28fa4c27504776bd4369fb26d77"}
Oct 30 01:29:46.837539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.837094    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx" event={"ID":"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33","Type":"ContainerStarted","Data":"8796ee10d85d8698bfe30b4701cad216ee37ad6425c7e6ae3eee906147d3a9b2"}
Oct 30 01:29:46.839142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.839095    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq" event={"ID":"f451ba8b-697a-4bea-88bf-b63e4e1c9690","Type":"ContainerStarted","Data":"f951d69962db72409b7f33dd06a6befa47bd13f3da72a4cfad3132b32e8e4450"}
Oct 30 01:29:46.839294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.839165    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq" event={"ID":"f451ba8b-697a-4bea-88bf-b63e4e1c9690","Type":"ContainerStarted","Data":"65ef6fbe3fd610094ccb615d3ff8aa4b5f7159df0b738515e053ebd2f3395a30"}
Oct 30 01:29:46.971884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.971817    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx" podStartSLOduration=1.9717942640000001 podStartE2EDuration="1.971794264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:46.927549107 +0000 UTC m=+2828.475239616" watchObservedRunningTime="2026-10-30 01:29:46.971794264 +0000 UTC m=+2828.519484762"
Oct 30 01:29:46.972052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:46.972011    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq" podStartSLOduration=1.972002688 podStartE2EDuration="1.972002688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:46.966826743 +0000 UTC m=+2828.514517254" watchObservedRunningTime="2026-10-30 01:29:46.972002688 +0000 UTC m=+2828.519693185"
Oct 30 01:29:47.090680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.090650    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:47.228971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.228934    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-kube-api-access-hs6pv\" (UniqueName: \"kubernetes.io/projected/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-kube-api-access-hs6pv\") pod \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") "
Oct 30 01:29:47.229170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.229100    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-openshift-ca\") pod \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") "
Oct 30 01:29:47.229246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.229222    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-xks-ca\") pod \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\" (UID: \"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb\") "
Oct 30 01:29:47.229584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.229548    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-openshift-ca" pod "4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" (UID: "4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:47.229816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.229794    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:47.231335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.231306    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-xks-ca" pod "4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" (UID: "4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:47.231522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.231376    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-kube-api-access-hs6pv" pod "4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" (UID: "4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb"). InnerVolumeSpecName "kube-api-access-hs6pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:47.330933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.330843    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:47.330933 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.330876    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6pv\" (UniqueName: \"kubernetes.io/projected/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb-kube-api-access-hs6pv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:47.850533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.850503    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" event={"ID":"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3","Type":"ContainerDied","Data":"6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511"}
Oct 30 01:29:47.850533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.850443    5104 generic.go:356] "Generic (PLEG): container finished" podID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerID="6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511" exitCode=0
Oct 30 01:29:47.851051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.850992    5104 scope.go:122] "RemoveContainer" containerID="6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511"
Oct 30 01:29:47.852487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.852440    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"
Oct 30 01:29:47.852632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.852470    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb" event={"ID":"4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb","Type":"ContainerDied","Data":"bf5efbb4102d0a9a02d9dacd0e00b9f860ce440b87ce0420257ac48d08026977"}
Oct 30 01:29:47.852632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.852624    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5efbb4102d0a9a02d9dacd0e00b9f860ce440b87ce0420257ac48d08026977"
Oct 30 01:29:47.852944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.852915    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:29:47.853054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.852984    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:29:47.853054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.853025    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:29:47.854435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.854411    5104 generic.go:356] "Generic (PLEG): container finished" podID="f451ba8b-697a-4bea-88bf-b63e4e1c9690" containerID="f951d69962db72409b7f33dd06a6befa47bd13f3da72a4cfad3132b32e8e4450" exitCode=0
Oct 30 01:29:47.854574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.854489    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq" event={"ID":"f451ba8b-697a-4bea-88bf-b63e4e1c9690","Type":"ContainerDied","Data":"f951d69962db72409b7f33dd06a6befa47bd13f3da72a4cfad3132b32e8e4450"}
Oct 30 01:29:47.856625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.856495    5104 generic.go:356] "Generic (PLEG): container finished" podID="67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" containerID="06fc7cb739ef626c6afadfa5493a0afad35bb28fa4c27504776bd4369fb26d77" exitCode=0
Oct 30 01:29:47.856625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:47.856569    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx" event={"ID":"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33","Type":"ContainerDied","Data":"06fc7cb739ef626c6afadfa5493a0afad35bb28fa4c27504776bd4369fb26d77"}
Oct 30 01:29:48.562326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:48.562284    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"]
Oct 30 01:29:48.562565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:48.562539    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy" containerID="cri-o://b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119" gracePeriod=30
Oct 30 01:29:48.567231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:48.567205    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"]
Oct 30 01:29:48.567439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:48.567413    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy" containerID="cri-o://85eee898b69c7e27059218d6eaba0cd16bf93aa2c0a3b885c5399fa317d173af" gracePeriod=30
Oct 30 01:29:48.582956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:48.582923    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"]
Oct 30 01:29:48.583171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:48.583143    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy" containerID="cri-o://8de4323acdc17376037bbf9b4767c3265f9069a99134005285d962ee99aab437" gracePeriod=30
Oct 30 01:29:48.894127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:48.893992    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" event={"ID":"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3","Type":"ContainerStarted","Data":"3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c"}
Oct 30 01:29:49.310885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.304671    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:29:49.320431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.316720    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 30 01:29:49.320431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.316791    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 30 01:29:49.323508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.322994    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:49.325508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.325147    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.390938    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-xks-ca\") pod \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") "
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.391000    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f451ba8b-697a-4bea-88bf-b63e4e1c9690-kube-api-access-m2nmx\" (UniqueName: \"kubernetes.io/projected/f451ba8b-697a-4bea-88bf-b63e4e1c9690-kube-api-access-m2nmx\") pod \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") "
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.391053    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-openshift-ca\") pod \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") "
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.391110    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-xks-ca\") pod \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\" (UID: \"f451ba8b-697a-4bea-88bf-b63e4e1c9690\") "
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.391346    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-kube-api-access-82vcs\" (UniqueName: \"kubernetes.io/projected/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-kube-api-access-82vcs\") pod \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") "
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.391476    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-openshift-ca\") pod \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\" (UID: \"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33\") "
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.394692    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-openshift-ca" pod "67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" (UID: "67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:49.395479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.395096    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-openshift-ca" pod "f451ba8b-697a-4bea-88bf-b63e4e1c9690" (UID: "f451ba8b-697a-4bea-88bf-b63e4e1c9690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:49.434261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.434213    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-xks-ca" pod "f451ba8b-697a-4bea-88bf-b63e4e1c9690" (UID: "f451ba8b-697a-4bea-88bf-b63e4e1c9690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:49.434647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.434594    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-xks-ca" pod "67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" (UID: "67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:49.437648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.437317    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-kube-api-access-82vcs" pod "67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" (UID: "67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33"). InnerVolumeSpecName "kube-api-access-82vcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:49.438287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.438255    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f451ba8b-697a-4bea-88bf-b63e4e1c9690-kube-api-access-m2nmx" pod "f451ba8b-697a-4bea-88bf-b63e4e1c9690" (UID: "f451ba8b-697a-4bea-88bf-b63e4e1c9690"). InnerVolumeSpecName "kube-api-access-m2nmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:49.511558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.493687    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:49.511558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.493740    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2nmx\" (UniqueName: \"kubernetes.io/projected/f451ba8b-697a-4bea-88bf-b63e4e1c9690-kube-api-access-m2nmx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:49.511558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.493758    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:49.511558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.493771    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451ba8b-697a-4bea-88bf-b63e4e1c9690-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:49.511558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.493786    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82vcs\" (UniqueName: \"kubernetes.io/projected/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-kube-api-access-82vcs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:49.511558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.493800    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:49.868540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.868445    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"]
Oct 30 01:29:49.921495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.917475    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"]
Oct 30 01:29:49.928387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.928355    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq" event={"ID":"f451ba8b-697a-4bea-88bf-b63e4e1c9690","Type":"ContainerDied","Data":"65ef6fbe3fd610094ccb615d3ff8aa4b5f7159df0b738515e053ebd2f3395a30"}
Oct 30 01:29:49.928647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.928410    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ef6fbe3fd610094ccb615d3ff8aa4b5f7159df0b738515e053ebd2f3395a30"
Oct 30 01:29:49.928647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.928525    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"
Oct 30 01:29:49.934522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.933739    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"
Oct 30 01:29:49.934522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.933811    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx" event={"ID":"67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33","Type":"ContainerDied","Data":"8796ee10d85d8698bfe30b4701cad216ee37ad6425c7e6ae3eee906147d3a9b2"}
Oct 30 01:29:49.934522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:49.933848    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8796ee10d85d8698bfe30b4701cad216ee37ad6425c7e6ae3eee906147d3a9b2"
Oct 30 01:29:50.303906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.303864    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 30 01:29:50.304141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.303929    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 30 01:29:50.465756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.465723    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"]
Oct 30 01:29:50.823904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.823869    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:29:50.832611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.832581    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-data\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.832855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.832841    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-credential-socket\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.832847    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-data" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.832977    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-credential-socket" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.832986    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-token\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833060    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-envoy\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833082    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1ca365-14ab-4b2f-a933-60387a9c687b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ca365-14ab-4b2f-a933-60387a9c687b-istiod-ca-cert\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833104    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-podinfo\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833139    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-socket\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833202    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-kube-api-access-t2hd9\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-kube-api-access-t2hd9\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833235    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-certs\") pod \"8e1ca365-14ab-4b2f-a933-60387a9c687b\" (UID: \"8e1ca365-14ab-4b2f-a933-60387a9c687b\") "
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833432    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-socket" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833510    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1ca365-14ab-4b2f-a933-60387a9c687b-istiod-ca-cert" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:50.833815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833693    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-certs" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:50.834327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833953    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.834327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833974    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.834327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.833988    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.834327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.834001    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ca365-14ab-4b2f-a933-60387a9c687b-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.834327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.834015    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.835157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.835133    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-token" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:50.835617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.835592    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-podinfo" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:50.835797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.835650    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-envoy" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:50.836229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.836193    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-kube-api-access-t2hd9" pod "8e1ca365-14ab-4b2f-a933-60387a9c687b" (UID: "8e1ca365-14ab-4b2f-a933-60387a9c687b"). InnerVolumeSpecName "kube-api-access-t2hd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:50.938698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.938659    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.939243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.939224    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.939341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.939330    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ca365-14ab-4b2f-a933-60387a9c687b-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.939425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.939413    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2hd9\" (UniqueName: \"kubernetes.io/projected/8e1ca365-14ab-4b2f-a933-60387a9c687b-kube-api-access-t2hd9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:50.968756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.968697    5104 generic.go:356] "Generic (PLEG): container finished" podID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerID="6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3" exitCode=0
Oct 30 01:29:50.969234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.969201    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"
Oct 30 01:29:50.972389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.969230    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" event={"ID":"8e1ca365-14ab-4b2f-a933-60387a9c687b","Type":"ContainerDied","Data":"6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3"}
Oct 30 01:29:50.972548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.972448    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j" event={"ID":"8e1ca365-14ab-4b2f-a933-60387a9c687b","Type":"ContainerDied","Data":"9eff245986fe750450271cbbc85b897a84039f945989ad137a0bde4e4d34904c"}
Oct 30 01:29:50.972548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:50.972520    5104 scope.go:122] "RemoveContainer" containerID="6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3"
Oct 30 01:29:51.029135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.029106    5104 scope.go:122] "RemoveContainer" containerID="6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3"
Oct 30 01:29:51.029572 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:51.029538    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3\": container with ID starting with 6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3 not found: ID does not exist" containerID="6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3"
Oct 30 01:29:51.029690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.029585    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3"} err="failed to get container status \"6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3\": rpc error: code = NotFound desc = could not find container \"6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3\": container with ID starting with 6c45d654155948cc987da5a01d79b6fbed15a0bb6e633499ee9542f16bff48e3 not found: ID does not exist"
Oct 30 01:29:51.063874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.063787    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"]
Oct 30 01:29:51.068122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.068095    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-nlq5j"]
Oct 30 01:29:51.129930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.129884    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:29:51.301491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.300891    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 30 01:29:51.301491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.300957    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 30 01:29:51.819257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.818706    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"]
Oct 30 01:29:51.819257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:51.818952    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy" containerID="cri-o://1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517" gracePeriod=30
Oct 30 01:29:52.111748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.111647    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"]
Oct 30 01:29:52.112412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112391    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f451ba8b-697a-4bea-88bf-b63e4e1c9690" containerName="revoke-keys"
Oct 30 01:29:52.112479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112414    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f451ba8b-697a-4bea-88bf-b63e4e1c9690" containerName="revoke-keys"
Oct 30 01:29:52.112525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112492    5104 cpu_manager.go:414] "containerMap: removing container" podUID="67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" containerName="revoke-keys"
Oct 30 01:29:52.112525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112502    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" containerName="revoke-keys"
Oct 30 01:29:52.112525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112517    5104 cpu_manager.go:414] "containerMap: removing container" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy"
Oct 30 01:29:52.112525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112523    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy"
Oct 30 01:29:52.112678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112530    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" containerName="revoke-keys"
Oct 30 01:29:52.112678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112535    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" containerName="revoke-keys"
Oct 30 01:29:52.112678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112656    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" containerName="revoke-keys"
Oct 30 01:29:52.112678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112665    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" containerName="istio-proxy"
Oct 30 01:29:52.112678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112674    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" containerName="revoke-keys"
Oct 30 01:29:52.112846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.112681    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f451ba8b-697a-4bea-88bf-b63e4e1c9690" containerName="revoke-keys"
Oct 30 01:29:52.118004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.117982    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.131102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.131064    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"]
Oct 30 01:29:52.132680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.132653    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:29:52.132812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.132713    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:52.135627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.135589    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"]
Oct 30 01:29:52.142000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.141971    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.159594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.156505    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dh4j\" (UniqueName: \"kubernetes.io/projected/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-kube-api-access-9dh4j\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.159594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.156647    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.159594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.156723    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfv9\" (UniqueName: \"kubernetes.io/projected/858f65ae-f849-4d29-9d85-865aa30e4352-kube-api-access-tdfv9\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.159594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.156781    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.159594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.156832    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.159594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.156879    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.165329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.165296    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"]
Oct 30 01:29:52.258899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.258843    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.259133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.258933    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.259133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.259062    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dh4j\" (UniqueName: \"kubernetes.io/projected/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-kube-api-access-9dh4j\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.259278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.259194    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.259278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.259269    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfv9\" (UniqueName: \"kubernetes.io/projected/858f65ae-f849-4d29-9d85-865aa30e4352-kube-api-access-tdfv9\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.259398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.259333    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.259852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.259820    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.260174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.260150    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.262514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.262029    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.262514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.262033    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.310482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.310197    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 30 01:29:52.310482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.310254    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 30 01:29:52.312419 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.312386    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfv9\" (UniqueName: \"kubernetes.io/projected/858f65ae-f849-4d29-9d85-865aa30e4352-kube-api-access-tdfv9\") pod \"maas-api-revoke-sub-54d046b7-q62q6\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.312616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.312386    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dh4j\" (UniqueName: \"kubernetes.io/projected/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-kube-api-access-9dh4j\") pod \"maas-api-revoke-sub-4e5c72e7-xt5c7\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.435784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.435749    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:52.464628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.464602    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:52.665714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.665469    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1ca365-14ab-4b2f-a933-60387a9c687b" path="/var/lib/kubelet/pods/8e1ca365-14ab-4b2f-a933-60387a9c687b/volumes"
Oct 30 01:29:52.919516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:52.919482    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"]
Oct 30 01:29:52.921420 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:29:52.921374    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858f65ae_f849_4d29_9d85_865aa30e4352.slice/crio-be73a392d139b37aba73a984c603fe55c8d5de1192d5f2947e28658f10124987 WatchSource:0}: Error finding container be73a392d139b37aba73a984c603fe55c8d5de1192d5f2947e28658f10124987: Status 404 returned error can't find the container with id be73a392d139b37aba73a984c603fe55c8d5de1192d5f2947e28658f10124987
Oct 30 01:29:53.009804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:53.009774    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"]
Oct 30 01:29:53.029858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:53.029829    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6" event={"ID":"858f65ae-f849-4d29-9d85-865aa30e4352","Type":"ContainerStarted","Data":"be73a392d139b37aba73a984c603fe55c8d5de1192d5f2947e28658f10124987"}
Oct 30 01:29:53.031728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:53.031704    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7" event={"ID":"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a","Type":"ContainerStarted","Data":"a5b71ea62cad94d484516da4a1d0cc5b06400a320384a93ccc3aa6f8fe9f0a51"}
Oct 30 01:29:53.309541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:53.307953    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 30 01:29:53.309541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:53.308009    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 30 01:29:54.020519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.020478    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:29:54.039000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.038966    5104 scope.go:122] "RemoveContainer" containerID="6c21fce2eb0dd5b090a9861568414eb61587273b2414a06804822145f9ae7322"
Oct 30 01:29:54.048343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.048304    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:29:54.054380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.054340    5104 generic.go:356] "Generic (PLEG): container finished" podID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerID="85eee898b69c7e27059218d6eaba0cd16bf93aa2c0a3b885c5399fa317d173af" exitCode=0
Oct 30 01:29:54.054573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.054549    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" event={"ID":"4f3a73a2-922c-4ecd-8e7d-4cde4f841835","Type":"ContainerDied","Data":"85eee898b69c7e27059218d6eaba0cd16bf93aa2c0a3b885c5399fa317d173af"}
Oct 30 01:29:54.054675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.054636    5104 scope.go:122] "RemoveContainer" containerID="6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9"
Oct 30 01:29:54.059830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.059719    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6" event={"ID":"858f65ae-f849-4d29-9d85-865aa30e4352","Type":"ContainerStarted","Data":"9b74ad9989b7351a1a76c7124c105704657d7f2c3c355c3ccbb3ab583f348f68"}
Oct 30 01:29:54.065764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.065729    5104 generic.go:356] "Generic (PLEG): container finished" podID="092373c6-de73-4075-9c94-340796cb9455" containerID="b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119" exitCode=0
Oct 30 01:29:54.066882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.066001    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"
Oct 30 01:29:54.066882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.066019    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" event={"ID":"092373c6-de73-4075-9c94-340796cb9455","Type":"ContainerDied","Data":"b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"}
Oct 30 01:29:54.066882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.066339    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz" event={"ID":"092373c6-de73-4075-9c94-340796cb9455","Type":"ContainerDied","Data":"726085de7ea523cfbec2cdeb55890636578c69ec671ce4e675253f47fb11fa96"}
Oct 30 01:29:54.071567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.070285    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7" event={"ID":"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a","Type":"ContainerStarted","Data":"a7d059ef4ba13cd176684c2a881ccbce9e7edfd143102dd9334cff6f993ab367"}
Oct 30 01:29:54.096642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.078123    5104 generic.go:356] "Generic (PLEG): container finished" podID="095dd611-ece6-4654-9154-7ae46b0560f2" containerID="8de4323acdc17376037bbf9b4767c3265f9069a99134005285d962ee99aab437" exitCode=0
Oct 30 01:29:54.096642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.078190    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" event={"ID":"095dd611-ece6-4654-9154-7ae46b0560f2","Type":"ContainerDied","Data":"8de4323acdc17376037bbf9b4767c3265f9069a99134005285d962ee99aab437"}
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.113137    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-kube-api-access-ccr2g\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-kube-api-access-ccr2g\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.113238    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-socket\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.113291    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092373c6-de73-4075-9c94-340796cb9455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092373c6-de73-4075-9c94-340796cb9455-istiod-ca-cert\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.113340    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-data\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.113376    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-certs\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.117553    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-istio-token\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-istio-token\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.117657    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-credential-socket\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.117808    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-envoy\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.117866    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/092373c6-de73-4075-9c94-340796cb9455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092373c6-de73-4075-9c94-340796cb9455-istio-podinfo\") pod \"092373c6-de73-4075-9c94-340796cb9455\" (UID: \"092373c6-de73-4075-9c94-340796cb9455\") "
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.123231    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-credential-socket" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.125180    5104 scope.go:122] "RemoveContainer" containerID="e8e48cfb0b7b93c05e12994297636b27d7d27633d95fc072a9f0818369b5278b"
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.125482    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-data" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.125809    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-socket" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.126131    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092373c6-de73-4075-9c94-340796cb9455-istiod-ca-cert" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.128558    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-certs" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.129129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.128894    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-istio-token" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:54.136161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.136129    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092373c6-de73-4075-9c94-340796cb9455-istio-podinfo" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:54.154733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.151822    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-envoy" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.154733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.151908    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-kube-api-access-ccr2g" pod "092373c6-de73-4075-9c94-340796cb9455" (UID: "092373c6-de73-4075-9c94-340796cb9455"). InnerVolumeSpecName "kube-api-access-ccr2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:54.157516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.155985    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7" podStartSLOduration=2.155966676 podStartE2EDuration="2.155966676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:54.150139604 +0000 UTC m=+2835.697830095" watchObservedRunningTime="2026-10-30 01:29:54.155966676 +0000 UTC m=+2835.703657176"
Oct 30 01:29:54.161623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.161584    5104 scope.go:122] "RemoveContainer" containerID="b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"
Oct 30 01:29:54.169158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.169134    5104 scope.go:122] "RemoveContainer" containerID="b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"
Oct 30 01:29:54.223132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223094    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223131    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092373c6-de73-4075-9c94-340796cb9455-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223148    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccr2g\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-kube-api-access-ccr2g\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223163    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223177    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092373c6-de73-4075-9c94-340796cb9455-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223190    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223203    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223217    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092373c6-de73-4075-9c94-340796cb9455-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.223358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.223229    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092373c6-de73-4075-9c94-340796cb9455-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.231693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.231194    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:29:54.231974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.231716    5104 scope.go:122] "RemoveContainer" containerID="8de4323acdc17376037bbf9b4767c3265f9069a99134005285d962ee99aab437"
Oct 30 01:29:54.233801 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:54.233767    5104 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz_openshift-ingress_092373c6-de73-4075-9c94-340796cb9455_0 in pod sandbox 726085de7ea523cfbec2cdeb55890636578c69ec671ce4e675253f47fb11fa96 from index: no such id: 'b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119'" containerID="b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"
Oct 30 01:29:54.234027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.233988    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"} err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz_openshift-ingress_092373c6-de73-4075-9c94-340796cb9455_0 in pod sandbox 726085de7ea523cfbec2cdeb55890636578c69ec671ce4e675253f47fb11fa96 from index: no such id: 'b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119'"
Oct 30 01:29:54.234159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.234145    5104 scope.go:122] "RemoveContainer" containerID="b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"
Oct 30 01:29:54.234932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.234792    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6" podStartSLOduration=2.234776193 podStartE2EDuration="2.234776193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:54.220898988 +0000 UTC m=+2835.768589484" watchObservedRunningTime="2026-10-30 01:29:54.234776193 +0000 UTC m=+2835.782466686"
Oct 30 01:29:54.234932 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:54.234859    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119\": container with ID starting with b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119 not found: ID does not exist" containerID="b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"
Oct 30 01:29:54.234932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.234893    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119"} err="failed to get container status \"b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119\": rpc error: code = NotFound desc = could not find container \"b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119\": container with ID starting with b45808477da29fd726c1923e1241a6a9c8f40c2b6e97b3170cf1ee3a654a0119 not found: ID does not exist"
Oct 30 01:29:54.263646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.263617    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:29:54.268437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.268419    5104 scope.go:122] "RemoveContainer" containerID="6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9"
Oct 30 01:29:54.268777 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:54.268756    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9\": container with ID starting with 6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9 not found: ID does not exist" containerID="6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9"
Oct 30 01:29:54.268864 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:54.268787    5104 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9\": rpc error: code = NotFound desc = could not find container \"6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9\": container with ID starting with 6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9 not found: ID does not exist" containerID="6b752b5bec2cd1d495d3f13ce8802bdde59046e55d9a9eee5c89a6d3947f8ec9"
Oct 30 01:29:54.327965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.306929    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 30 01:29:54.327965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.306984    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 30 01:29:54.441207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441093    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-kube-api-access-d2h5f\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-kube-api-access-d2h5f\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.441207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441161    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-podinfo\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.441207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441202    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-data\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.441442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441326    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-socket\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.441442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441390    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istiod-ca-cert\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.441592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441527    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-certs\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.441592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441578    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-envoy\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.441715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441634    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-istio-token\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-istio-token\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.441715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441680    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095dd611-ece6-4654-9154-7ae46b0560f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095dd611-ece6-4654-9154-7ae46b0560f2-istiod-ca-cert\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.441841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441746    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-kube-api-access-x4w2n\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-kube-api-access-x4w2n\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.441841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441796    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-credential-socket\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.441841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441821    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-token\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.442022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441853    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-envoy\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.442022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441913    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-socket\") pod \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\" (UID: \"4f3a73a2-922c-4ecd-8e7d-4cde4f841835\") "
Oct 30 01:29:54.442022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.441959    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-credential-socket\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.442022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.442016    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-data\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.442248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.442105    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-certs\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.442248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.442175    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/095dd611-ece6-4654-9154-7ae46b0560f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095dd611-ece6-4654-9154-7ae46b0560f2-istio-podinfo\") pod \"095dd611-ece6-4654-9154-7ae46b0560f2\" (UID: \"095dd611-ece6-4654-9154-7ae46b0560f2\") "
Oct 30 01:29:54.446248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.446069    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-istio-token" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:54.446633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.446508    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istiod-ca-cert" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:54.446815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.446780    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-certs" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.447412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.447389    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-socket" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.447856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.447833    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-data" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.452090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.451848    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-envoy" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.452551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.452366    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095dd611-ece6-4654-9154-7ae46b0560f2-istiod-ca-cert" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:54.457807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.457652    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-data" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.457918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.457832    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-credential-socket" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.458877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.458551    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-credential-socket" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.459229 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.459186    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-kube-api-access-x4w2n" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "kube-api-access-x4w2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:54.459620 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.459570    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-certs" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.460619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460438    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-socket" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.460916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460830    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.460916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460858    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095dd611-ece6-4654-9154-7ae46b0560f2-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.460916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460872    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4w2n\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-kube-api-access-x4w2n\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.460916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460886    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.460916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460898    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.460916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460911    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.461300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460924    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.461300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460935    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.461300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460948    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.461300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460960    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.461300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460972    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095dd611-ece6-4654-9154-7ae46b0560f2-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.461300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.460987    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.461300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.461000    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.462058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.462022    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-envoy" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:54.463099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.462940    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-token" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:54.463099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.463070    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-kube-api-access-d2h5f" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "kube-api-access-d2h5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:54.466628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.466603    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-podinfo" pod "4f3a73a2-922c-4ecd-8e7d-4cde4f841835" (UID: "4f3a73a2-922c-4ecd-8e7d-4cde4f841835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:54.467594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.467574    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"]
Oct 30 01:29:54.468640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.468606    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/095dd611-ece6-4654-9154-7ae46b0560f2-istio-podinfo" pod "095dd611-ece6-4654-9154-7ae46b0560f2" (UID: "095dd611-ece6-4654-9154-7ae46b0560f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:54.473930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.473901    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-9ljxz"]
Oct 30 01:29:54.564301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.564270    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.564618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.564489    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.564618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.564510    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095dd611-ece6-4654-9154-7ae46b0560f2-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.564618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.564526    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2h5f\" (UniqueName: \"kubernetes.io/projected/095dd611-ece6-4654-9154-7ae46b0560f2-kube-api-access-d2h5f\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.564618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.564541    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3a73a2-922c-4ecd-8e7d-4cde4f841835-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.586869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:54.586817    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092373c6-de73-4075-9c94-340796cb9455" path="/var/lib/kubelet/pods/092373c6-de73-4075-9c94-340796cb9455/volumes"
Oct 30 01:29:55.089421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.089327    5104 generic.go:356] "Generic (PLEG): container finished" podID="3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" containerID="a7d059ef4ba13cd176684c2a881ccbce9e7edfd143102dd9334cff6f993ab367" exitCode=0
Oct 30 01:29:55.089638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.089422    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7" event={"ID":"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a","Type":"ContainerDied","Data":"a7d059ef4ba13cd176684c2a881ccbce9e7edfd143102dd9334cff6f993ab367"}
Oct 30 01:29:55.089638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.089505    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t" event={"ID":"095dd611-ece6-4654-9154-7ae46b0560f2","Type":"ContainerDied","Data":"d774c4c0eeeb6d0cf39112a1e6677337008cdb8b5c7505406fb41bf64f895195"}
Oct 30 01:29:55.091433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.091410    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"
Oct 30 01:29:55.091612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.091439    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4" event={"ID":"4f3a73a2-922c-4ecd-8e7d-4cde4f841835","Type":"ContainerDied","Data":"2adcd8be0d23214bbcaeda18530e552e851390fb6e7daa506276fa403ba3d2bf"}
Oct 30 01:29:55.091612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.091540    5104 scope.go:122] "RemoveContainer" containerID="85eee898b69c7e27059218d6eaba0cd16bf93aa2c0a3b885c5399fa317d173af"
Oct 30 01:29:55.093704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.093678    5104 generic.go:356] "Generic (PLEG): container finished" podID="858f65ae-f849-4d29-9d85-865aa30e4352" containerID="9b74ad9989b7351a1a76c7124c105704657d7f2c3c355c3ccbb3ab583f348f68" exitCode=0
Oct 30 01:29:55.093839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.093753    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6" event={"ID":"858f65ae-f849-4d29-9d85-865aa30e4352","Type":"ContainerDied","Data":"9b74ad9989b7351a1a76c7124c105704657d7f2c3c355c3ccbb3ab583f348f68"}
Oct 30 01:29:55.093979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.093962    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"
Oct 30 01:29:55.232040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.231997    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"]
Oct 30 01:29:55.237218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.237182    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-sxg6t"]
Oct 30 01:29:55.270786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.270751    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"]
Oct 30 01:29:55.277045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.277005    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tc5q4"]
Oct 30 01:29:55.305178 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.305142    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:29:55.305584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.305557    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:29:55.306346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.306318    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:29:55.534594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.534560    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"]
Oct 30 01:29:55.573235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:55.573199    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"]
Oct 30 01:29:56.441796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.441768    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:56.443577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.443549    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"]
Oct 30 01:29:56.483018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.482486    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"]
Oct 30 01:29:56.507508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.507414    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-xks-ca\") pod \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") "
Oct 30 01:29:56.507704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.507575    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-kube-api-access-9dh4j\" (UniqueName: \"kubernetes.io/projected/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-kube-api-access-9dh4j\") pod \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") "
Oct 30 01:29:56.518512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.509548    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-openshift-ca\") pod \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\" (UID: \"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a\") "
Oct 30 01:29:56.518512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.510810    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-openshift-ca" pod "3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" (UID: "3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:56.518512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.514149    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-kube-api-access-9dh4j" pod "3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" (UID: "3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a"). InnerVolumeSpecName "kube-api-access-9dh4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:56.522136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.522087    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-xks-ca" pod "3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" (UID: "3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:56.525146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.525120    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:56.597889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.597849    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" path="/var/lib/kubelet/pods/095dd611-ece6-4654-9154-7ae46b0560f2/volumes"
Oct 30 01:29:56.599287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.599256    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" path="/var/lib/kubelet/pods/4f3a73a2-922c-4ecd-8e7d-4cde4f841835/volumes"
Oct 30 01:29:56.612672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.612639    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:56.612672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.612674    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dh4j\" (UniqueName: \"kubernetes.io/projected/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-kube-api-access-9dh4j\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:56.612910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.612690    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:56.713639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.713594    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858f65ae-f849-4d29-9d85-865aa30e4352-kube-api-access-tdfv9\" (UniqueName: \"kubernetes.io/projected/858f65ae-f849-4d29-9d85-865aa30e4352-kube-api-access-tdfv9\") pod \"858f65ae-f849-4d29-9d85-865aa30e4352\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") "
Oct 30 01:29:56.713827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.713670    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-xks-ca\") pod \"858f65ae-f849-4d29-9d85-865aa30e4352\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") "
Oct 30 01:29:56.713827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.713808    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-openshift-ca\") pod \"858f65ae-f849-4d29-9d85-865aa30e4352\" (UID: \"858f65ae-f849-4d29-9d85-865aa30e4352\") "
Oct 30 01:29:56.714283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.714253    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-openshift-ca" pod "858f65ae-f849-4d29-9d85-865aa30e4352" (UID: "858f65ae-f849-4d29-9d85-865aa30e4352"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:56.714603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.714579    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:56.715917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.715891    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-xks-ca" pod "858f65ae-f849-4d29-9d85-865aa30e4352" (UID: "858f65ae-f849-4d29-9d85-865aa30e4352"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:56.716028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.716006    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858f65ae-f849-4d29-9d85-865aa30e4352-kube-api-access-tdfv9" pod "858f65ae-f849-4d29-9d85-865aa30e4352" (UID: "858f65ae-f849-4d29-9d85-865aa30e4352"). InnerVolumeSpecName "kube-api-access-tdfv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:56.815997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.815959    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858f65ae-f849-4d29-9d85-865aa30e4352-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:56.815997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:56.816000    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdfv9\" (UniqueName: \"kubernetes.io/projected/858f65ae-f849-4d29-9d85-865aa30e4352-kube-api-access-tdfv9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.085967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.085935    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:29:57.117000 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.116959    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6" event={"ID":"858f65ae-f849-4d29-9d85-865aa30e4352","Type":"ContainerDied","Data":"be73a392d139b37aba73a984c603fe55c8d5de1192d5f2947e28658f10124987"}
Oct 30 01:29:57.117242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.117062    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be73a392d139b37aba73a984c603fe55c8d5de1192d5f2947e28658f10124987"
Oct 30 01:29:57.117417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.117389    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120365    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-socket\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120514    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-data\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120578    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-credential-socket\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120600    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7" event={"ID":"3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a","Type":"ContainerDied","Data":"a5b71ea62cad94d484516da4a1d0cc5b06400a320384a93ccc3aa6f8fe9f0a51"}
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120652    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b71ea62cad94d484516da4a1d0cc5b06400a320384a93ccc3aa6f8fe9f0a51"
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120611    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-certs\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120739    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-envoy\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120830    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-token\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-token\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120905    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-certs" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.120934    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-podinfo\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121013    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istiod-ca-cert\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121072    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-kube-api-access-t8p9k\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-kube-api-access-t8p9k\") pod \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\" (UID: \"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c\") "
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121200    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-socket" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121402    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-data" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121702    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121719    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.121776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121758    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.122896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.121969    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-credential-socket" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:57.123627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.123558    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istiod-ca-cert" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:57.123627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.123596    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"
Oct 30 01:29:57.133602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.126142    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-envoy" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:29:57.133602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.128664    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-podinfo" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:29:57.204054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.203815    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-token" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:57.206198 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.206163    5104 generic.go:356] "Generic (PLEG): container finished" podID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerID="1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517" exitCode=0
Oct 30 01:29:57.210246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.206476    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"
Oct 30 01:29:57.210246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.206498    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" event={"ID":"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c","Type":"ContainerDied","Data":"1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517"}
Oct 30 01:29:57.210246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.210050    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v" event={"ID":"52829d0c-ebfe-4b80-b611-c3b42b1c0a2c","Type":"ContainerDied","Data":"db28d5df2e13851ce0fa698617781b00b213db6959ac0aaeedb14fec30b79b3c"}
Oct 30 01:29:57.210246 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.210087    5104 scope.go:122] "RemoveContainer" containerID="1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517"
Oct 30 01:29:57.214784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.214675    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-kube-api-access-t8p9k" pod "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" (UID: "52829d0c-ebfe-4b80-b611-c3b42b1c0a2c"). InnerVolumeSpecName "kube-api-access-t8p9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:57.224852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.223617    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.224852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.223651    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.224852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.223666    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.224852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.223680    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.224852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.223696    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.224852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.223723    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8p9k\" (UniqueName: \"kubernetes.io/projected/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c-kube-api-access-t8p9k\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:29:57.243947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.243819    5104 scope.go:122] "RemoveContainer" containerID="1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517"
Oct 30 01:29:57.244309 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:29:57.244219    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517\": container with ID starting with 1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517 not found: ID does not exist" containerID="1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517"
Oct 30 01:29:57.244309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.244260    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517"} err="failed to get container status \"1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517\": rpc error: code = NotFound desc = could not find container \"1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517\": container with ID starting with 1860ecf45adb0433043bf47efab2e470867adefbfafe023323dd78be6e975517 not found: ID does not exist"
Oct 30 01:29:57.558445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.558405    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"]
Oct 30 01:29:57.568162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:57.568124    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-dh45v"]
Oct 30 01:29:58.580471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:58.580426    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" path="/var/lib/kubelet/pods/52829d0c-ebfe-4b80-b611-c3b42b1c0a2c/volumes"
Oct 30 01:29:58.755866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:58.755826    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"]
Oct 30 01:29:58.766989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:29:58.766957    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-29145c-plzbc"]
Oct 30 01:30:00.154031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.153998    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"]
Oct 30 01:30:00.154766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154747    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" containerName="revoke-keys"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154768    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" containerName="revoke-keys"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154779    5104 cpu_manager.go:414] "containerMap: removing container" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154785    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154796    5104 cpu_manager.go:414] "containerMap: removing container" podUID="858f65ae-f849-4d29-9d85-865aa30e4352" containerName="revoke-keys"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154802    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858f65ae-f849-4d29-9d85-865aa30e4352" containerName="revoke-keys"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154809    5104 cpu_manager.go:414] "containerMap: removing container" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154814    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy"
Oct 30 01:30:00.154820 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154823    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154828    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154835    5104 cpu_manager.go:414] "containerMap: removing container" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154840    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154856    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154860    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154971    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" containerName="revoke-keys"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154983    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="52829d0c-ebfe-4b80-b611-c3b42b1c0a2c" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154992    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="858f65ae-f849-4d29-9d85-865aa30e4352" containerName="revoke-keys"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.154999    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="092373c6-de73-4075-9c94-340796cb9455" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.155004    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.155012    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="095dd611-ece6-4654-9154-7ae46b0560f2" containerName="istio-proxy"
Oct 30 01:30:00.155191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.155021    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3a73a2-922c-4ecd-8e7d-4cde4f841835" containerName="istio-proxy"
Oct 30 01:30:00.160081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.160058    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"]
Oct 30 01:30:00.160224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.160183    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.164645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.164609    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"]
Oct 30 01:30:00.165849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.165826    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.182853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.182825    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:30:00.188030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.187997    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"]
Oct 30 01:30:00.188610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.188588    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.197756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.197728    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"]
Oct 30 01:30:00.198427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.198158    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.210073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.210041    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"]
Oct 30 01:30:00.210736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.210710    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.236134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.233165    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"]
Oct 30 01:30:00.236134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.233215    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"]
Oct 30 01:30:00.236134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.233244    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"]
Oct 30 01:30:00.236807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.236781    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.269749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.269711    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"]
Oct 30 01:30:00.269908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.269778    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"]
Oct 30 01:30:00.269908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.269805    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"]
Oct 30 01:30:00.269908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.269829    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"]
Oct 30 01:30:00.269908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.269859    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"]
Oct 30 01:30:00.270349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.270325    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.277897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.277859    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.278678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.278650    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdxg\" (UniqueName: \"kubernetes.io/projected/96a1808d-54fb-4bdc-865b-e293422c0d29-kube-api-access-7pdxg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.278994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.278974    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a1808d-54fb-4bdc-865b-e293422c0d29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.279240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.279220    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.279405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.279389    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75g5\" (UniqueName: \"kubernetes.io/projected/1969249d-ee44-4685-86de-2a77d4f02ba8-kube-api-access-b75g5\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.279613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.279596    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.279772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.279758    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a0b7f7-8c25-4635-8da5-efba3220e87a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.279931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.279916    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx9hn\" (UniqueName: \"kubernetes.io/projected/42a0b7f7-8c25-4635-8da5-efba3220e87a-kube-api-access-fx9hn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.281167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.281149    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhc7\" (UniqueName: \"kubernetes.io/projected/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-kube-api-access-shhc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.281353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.281338    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzww7\" (UniqueName: \"kubernetes.io/projected/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-kube-api-access-mzww7\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.281588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.281569    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f44k2\" (UniqueName: \"kubernetes.io/projected/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-kube-api-access-f44k2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.281825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.281806    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1969249d-ee44-4685-86de-2a77d4f02ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.283536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.283512    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"]
Oct 30 01:30:00.283623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.283593    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"]
Oct 30 01:30:00.285575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.285559    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:00.294886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.294858    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.308095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.308066    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"]
Oct 30 01:30:00.318831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.318806    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"]
Oct 30 01:30:00.383725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.383680    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.383954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.383746    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2r85\" (UniqueName: \"kubernetes.io/projected/b684507c-883f-403e-9090-16053c2b83fe-kube-api-access-n2r85\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.383954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.383784    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a0b7f7-8c25-4635-8da5-efba3220e87a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.383954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.383873    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx9hn\" (UniqueName: \"kubernetes.io/projected/42a0b7f7-8c25-4635-8da5-efba3220e87a-kube-api-access-fx9hn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.383954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.383906    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhc7\" (UniqueName: \"kubernetes.io/projected/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-kube-api-access-shhc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.383954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.383943    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzww7\" (UniqueName: \"kubernetes.io/projected/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-kube-api-access-mzww7\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.384264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.383985    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f44k2\" (UniqueName: \"kubernetes.io/projected/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-kube-api-access-f44k2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.384264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384032    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b684507c-883f-403e-9090-16053c2b83fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.384264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384066    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73745591-3ee2-4e22-b43e-453ce48e765a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.384264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384135    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98kmz\" (UniqueName: \"kubernetes.io/projected/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-kube-api-access-98kmz\") pod \"maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:00.384264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384204    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1969249d-ee44-4685-86de-2a77d4f02ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.384607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384398    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr69t\" (UniqueName: \"kubernetes.io/projected/73745591-3ee2-4e22-b43e-453ce48e765a-kube-api-access-lr69t\") pod \"maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.384607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384481    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.384607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384539    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a0b7f7-8c25-4635-8da5-efba3220e87a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.384607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384581    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdxg\" (UniqueName: \"kubernetes.io/projected/96a1808d-54fb-4bdc-865b-e293422c0d29-kube-api-access-7pdxg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.384837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384616    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.384837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384648    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a1808d-54fb-4bdc-865b-e293422c0d29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.384837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384708    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:00.384837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384744    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.384837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.384778    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75g5\" (UniqueName: \"kubernetes.io/projected/1969249d-ee44-4685-86de-2a77d4f02ba8-kube-api-access-b75g5\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.385126 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.385057    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1969249d-ee44-4685-86de-2a77d4f02ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.385197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.385160    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.385398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.385378    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.385513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.385380    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a1808d-54fb-4bdc-865b-e293422c0d29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.404342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.404274    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx9hn\" (UniqueName: \"kubernetes.io/projected/42a0b7f7-8c25-4635-8da5-efba3220e87a-kube-api-access-fx9hn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.404342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.404272    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhc7\" (UniqueName: \"kubernetes.io/projected/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-kube-api-access-shhc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.404972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.404947    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzww7\" (UniqueName: \"kubernetes.io/projected/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-kube-api-access-mzww7\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.405448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.405426    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f44k2\" (UniqueName: \"kubernetes.io/projected/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-kube-api-access-f44k2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.405582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.405551    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdxg\" (UniqueName: \"kubernetes.io/projected/96a1808d-54fb-4bdc-865b-e293422c0d29-kube-api-access-7pdxg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.406925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.406902    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75g5\" (UniqueName: \"kubernetes.io/projected/1969249d-ee44-4685-86de-2a77d4f02ba8-kube-api-access-b75g5\") pod \"maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.486628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.486584    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98kmz\" (UniqueName: \"kubernetes.io/projected/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-kube-api-access-98kmz\") pod \"maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:00.486852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.486689    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr69t\" (UniqueName: \"kubernetes.io/projected/73745591-3ee2-4e22-b43e-453ce48e765a-kube-api-access-lr69t\") pod \"maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.486852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.486814    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:00.487003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.486884    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r85\" (UniqueName: \"kubernetes.io/projected/b684507c-883f-403e-9090-16053c2b83fe-kube-api-access-n2r85\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.487067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.487001    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b684507c-883f-403e-9090-16053c2b83fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.487067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.487031    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73745591-3ee2-4e22-b43e-453ce48e765a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.487608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.487577    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:00.487733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.487696    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b684507c-883f-403e-9090-16053c2b83fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.487733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.487719    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73745591-3ee2-4e22-b43e-453ce48e765a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.496665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.496434    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:00.499186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.499162    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:00.506618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.506571    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr69t\" (UniqueName: \"kubernetes.io/projected/73745591-3ee2-4e22-b43e-453ce48e765a-kube-api-access-lr69t\") pod \"maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.506993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.506969    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98kmz\" (UniqueName: \"kubernetes.io/projected/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-kube-api-access-98kmz\") pod \"maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:00.506993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.506993    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r85\" (UniqueName: \"kubernetes.io/projected/b684507c-883f-403e-9090-16053c2b83fe-kube-api-access-n2r85\") pod \"maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.535040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.534998    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" podUID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" containerName="payload-processing" containerID="cri-o://be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac" gracePeriod=25
Oct 30 01:30:00.564323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.563943    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:00.569212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.569171    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:00.574286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.573560    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" podUID="7aab52d9-16c4-470f-97ac-6adbf20090ee" containerName="payload-pre-processing" containerID="cri-o://59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c" gracePeriod=25
Oct 30 01:30:00.577638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.577608    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:00.598960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.598727    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:00.600436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.600401    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2530c279-91a4-4f4f-92b2-29f21d4247d4" path="/var/lib/kubelet/pods/2530c279-91a4-4f4f-92b2-29f21d4247d4/volumes"
Oct 30 01:30:00.686447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.686216    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:00.686447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.686484    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:00.686814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:00.686590    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:01.025744 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.025707    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"]
Oct 30 01:30:01.028125 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.028091    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51b9f4c_2235_4095_860f_c0e5f8b7cd8c.slice/crio-7d9d4bf4dd1cbc3990730019f6b815f2d0473f640b507a4df67b2943f9827b3d WatchSource:0}: Error finding container 7d9d4bf4dd1cbc3990730019f6b815f2d0473f640b507a4df67b2943f9827b3d: Status 404 returned error can't find the container with id 7d9d4bf4dd1cbc3990730019f6b815f2d0473f640b507a4df67b2943f9827b3d
Oct 30 01:30:01.048766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.048731    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"]
Oct 30 01:30:01.049544 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.049512    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a1808d_54fb_4bdc_865b_e293422c0d29.slice/crio-e358d0513b474d05bc1c7c83454f547b746242c1431821c52a60ca92468f82f2 WatchSource:0}: Error finding container e358d0513b474d05bc1c7c83454f547b746242c1431821c52a60ca92468f82f2: Status 404 returned error can't find the container with id e358d0513b474d05bc1c7c83454f547b746242c1431821c52a60ca92468f82f2
Oct 30 01:30:01.129734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.129693    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:01.154915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.154806    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"]
Oct 30 01:30:01.157429 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.156585    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5e8754_fd77_46a6_b7a6_d933757dc0a2.slice/crio-2cc7f9bae5c973f17cf893abad316aaea8578a7d8692b7ec302687fb0a78ff89 WatchSource:0}: Error finding container 2cc7f9bae5c973f17cf893abad316aaea8578a7d8692b7ec302687fb0a78ff89: Status 404 returned error can't find the container with id 2cc7f9bae5c973f17cf893abad316aaea8578a7d8692b7ec302687fb0a78ff89
Oct 30 01:30:01.158531 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.158448    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73745591_3ee2_4e22_b43e_453ce48e765a.slice/crio-fd510f1e819b283fcf3bdb97bd7db7ebbc7a1707ae0be5584038f75288f7e668 WatchSource:0}: Error finding container fd510f1e819b283fcf3bdb97bd7db7ebbc7a1707ae0be5584038f75288f7e668: Status 404 returned error can't find the container with id fd510f1e819b283fcf3bdb97bd7db7ebbc7a1707ae0be5584038f75288f7e668
Oct 30 01:30:01.160421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.160371    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"]
Oct 30 01:30:01.169842 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.166576    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"]
Oct 30 01:30:01.169842 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.168154    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0b81d4_dd2c_4140_8ad3_e13a4da9d840.slice/crio-4e8b4efbfd25ce3f5b0df2894c0f7f69ec573f4f1d822d7e3862f82322c4b1ab WatchSource:0}: Error finding container 4e8b4efbfd25ce3f5b0df2894c0f7f69ec573f4f1d822d7e3862f82322c4b1ab: Status 404 returned error can't find the container with id 4e8b4efbfd25ce3f5b0df2894c0f7f69ec573f4f1d822d7e3862f82322c4b1ab
Oct 30 01:30:01.169842 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.169662    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1969249d_ee44_4685_86de_2a77d4f02ba8.slice/crio-2312f27244ea0c64a4a9e11962dc15b0b72c51dbec55cf67edb8cfa2628d1f3e WatchSource:0}: Error finding container 2312f27244ea0c64a4a9e11962dc15b0b72c51dbec55cf67edb8cfa2628d1f3e: Status 404 returned error can't find the container with id 2312f27244ea0c64a4a9e11962dc15b0b72c51dbec55cf67edb8cfa2628d1f3e
Oct 30 01:30:01.171490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.171432    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"]
Oct 30 01:30:01.235333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.235299    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"]
Oct 30 01:30:01.239418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.239396    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"]
Oct 30 01:30:01.267099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.267050    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" event={"ID":"73745591-3ee2-4e22-b43e-453ce48e765a","Type":"ContainerStarted","Data":"fd510f1e819b283fcf3bdb97bd7db7ebbc7a1707ae0be5584038f75288f7e668"}
Oct 30 01:30:01.268739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.268705    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" event={"ID":"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840","Type":"ContainerStarted","Data":"4e8b4efbfd25ce3f5b0df2894c0f7f69ec573f4f1d822d7e3862f82322c4b1ab"}
Oct 30 01:30:01.270768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.270734    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" event={"ID":"96a1808d-54fb-4bdc-865b-e293422c0d29","Type":"ContainerStarted","Data":"e358d0513b474d05bc1c7c83454f547b746242c1431821c52a60ca92468f82f2"}
Oct 30 01:30:01.272250 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.272220    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" event={"ID":"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c","Type":"ContainerStarted","Data":"7d9d4bf4dd1cbc3990730019f6b815f2d0473f640b507a4df67b2943f9827b3d"}
Oct 30 01:30:01.275990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.275949    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerStarted","Data":"2312f27244ea0c64a4a9e11962dc15b0b72c51dbec55cf67edb8cfa2628d1f3e"}
Oct 30 01:30:01.278558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.278492    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" event={"ID":"6f5e8754-fd77-46a6-b7a6-d933757dc0a2","Type":"ContainerStarted","Data":"2cc7f9bae5c973f17cf893abad316aaea8578a7d8692b7ec302687fb0a78ff89"}
Oct 30 01:30:01.298446 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.298411    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079d8eee_d0c4_43c5_89cc_31dc5c6d5ea8.slice/crio-293e3de6ee7d5658ea24433c10c82f9f7935e33bbb0cd5f85c62b7730440ba39 WatchSource:0}: Error finding container 293e3de6ee7d5658ea24433c10c82f9f7935e33bbb0cd5f85c62b7730440ba39: Status 404 returned error can't find the container with id 293e3de6ee7d5658ea24433c10c82f9f7935e33bbb0cd5f85c62b7730440ba39
Oct 30 01:30:01.303684 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.303638    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a0b7f7_8c25_4635_8da5_efba3220e87a.slice/crio-169023eef41789a79532bbe7592d1f92c978cb1fb68d545e9ca7732b570a727b WatchSource:0}: Error finding container 169023eef41789a79532bbe7592d1f92c978cb1fb68d545e9ca7732b570a727b: Status 404 returned error can't find the container with id 169023eef41789a79532bbe7592d1f92c978cb1fb68d545e9ca7732b570a727b
Oct 30 01:30:01.360963 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:01.360921    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb684507c_883f_403e_9090_16053c2b83fe.slice/crio-ef46ed4be5e89b53f98b06f4a9069f5a1bcf875cf0819adb7f4fab95d1b16216 WatchSource:0}: Error finding container ef46ed4be5e89b53f98b06f4a9069f5a1bcf875cf0819adb7f4fab95d1b16216: Status 404 returned error can't find the container with id ef46ed4be5e89b53f98b06f4a9069f5a1bcf875cf0819adb7f4fab95d1b16216
Oct 30 01:30:01.371782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.371751    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"]
Oct 30 01:30:01.444700 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.444498    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerName="payload-processing" containerID="cri-o://db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302" gracePeriod=25
Oct 30 01:30:01.490034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:01.483594    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" podUID="801aa334-19ec-4ae7-9c34-19bab145c954" containerName="payload-pre-processing" containerID="cri-o://2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b" gracePeriod=25
Oct 30 01:30:02.298876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.298828    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" event={"ID":"73745591-3ee2-4e22-b43e-453ce48e765a","Type":"ContainerStarted","Data":"ea28945302a8c9e6c942c76b89bebd60d0fd51643ab13841fe9cb4978bb2b31b"}
Oct 30 01:30:02.300926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.300892    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" event={"ID":"b684507c-883f-403e-9090-16053c2b83fe","Type":"ContainerStarted","Data":"4242b41dc7af0f890daa41c6bf4f6efc74f90567179a2f36ddf35ee21ff72a02"}
Oct 30 01:30:02.301098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.300952    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" event={"ID":"b684507c-883f-403e-9090-16053c2b83fe","Type":"ContainerStarted","Data":"ef46ed4be5e89b53f98b06f4a9069f5a1bcf875cf0819adb7f4fab95d1b16216"}
Oct 30 01:30:02.302841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.302807    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" event={"ID":"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840","Type":"ContainerStarted","Data":"bf045b0029c4e472fc7329c062e374ce41f4e029b48f995b27cf2ff473ff1559"}
Oct 30 01:30:02.304736 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.304709    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" event={"ID":"42a0b7f7-8c25-4635-8da5-efba3220e87a","Type":"ContainerStarted","Data":"7ae9ffd1ea652791b3c1d609c9b12a7b1d0205ae6678da406f5f992033e94414"}
Oct 30 01:30:02.304846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.304757    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" event={"ID":"42a0b7f7-8c25-4635-8da5-efba3220e87a","Type":"ContainerStarted","Data":"169023eef41789a79532bbe7592d1f92c978cb1fb68d545e9ca7732b570a727b"}
Oct 30 01:30:02.306331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.306306    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerStarted","Data":"2b1186b6aca36a30a367ab3e4d38644be0d1a254b1cb1b7b8d8e90cb4b1c9199"}
Oct 30 01:30:02.306466 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.306355    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerStarted","Data":"293e3de6ee7d5658ea24433c10c82f9f7935e33bbb0cd5f85c62b7730440ba39"}
Oct 30 01:30:02.307994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.307963    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" event={"ID":"96a1808d-54fb-4bdc-865b-e293422c0d29","Type":"ContainerStarted","Data":"a357b1f4f7144b64965322472f770354d302ef3c6daf2f502802f71bd92412e9"}
Oct 30 01:30:02.308251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.308234    5104 scope.go:122] "RemoveContainer" containerID="a357b1f4f7144b64965322472f770354d302ef3c6daf2f502802f71bd92412e9"
Oct 30 01:30:02.309836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.309800    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" event={"ID":"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c","Type":"ContainerStarted","Data":"dabcd91e0df518c403b9af2585849b7560c71c39e930dbd0b2eefb71c4b3cc11"}
Oct 30 01:30:02.310167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.310080    5104 scope.go:122] "RemoveContainer" containerID="dabcd91e0df518c403b9af2585849b7560c71c39e930dbd0b2eefb71c4b3cc11"
Oct 30 01:30:02.313428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.313386    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerStarted","Data":"2c1c652c55c1afa8bc80091ad02b482b466e39e384a471ce72448a1e730295c7"}
Oct 30 01:30:02.315512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.315480    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" event={"ID":"6f5e8754-fd77-46a6-b7a6-d933757dc0a2","Type":"ContainerStarted","Data":"9f5ce9e9493a59177200da885f2aa70fa7c1b8e4b25707a6aaa78f239df06327"}
Oct 30 01:30:02.335832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.335759    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" podStartSLOduration=2.335740876 podStartE2EDuration="2.335740876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.33449815 +0000 UTC m=+2843.882188643" watchObservedRunningTime="2026-10-30 01:30:02.335740876 +0000 UTC m=+2843.883431374"
Oct 30 01:30:02.389257 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.389182    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" podStartSLOduration=2.389162683 podStartE2EDuration="2.389162683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.378542408 +0000 UTC m=+2843.926232907" watchObservedRunningTime="2026-10-30 01:30:02.389162683 +0000 UTC m=+2843.936853179"
Oct 30 01:30:02.424534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.424450    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" podStartSLOduration=2.424426249 podStartE2EDuration="2.424426249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.417551136 +0000 UTC m=+2843.965241642" watchObservedRunningTime="2026-10-30 01:30:02.424426249 +0000 UTC m=+2843.972116746"
Oct 30 01:30:02.457488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.457429    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" podStartSLOduration=2.4574140509999998 podStartE2EDuration="2.457414051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.456048923 +0000 UTC m=+2844.003739419" watchObservedRunningTime="2026-10-30 01:30:02.457414051 +0000 UTC m=+2844.005104548"
Oct 30 01:30:02.616432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.615307    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" podStartSLOduration=2.615286164 podStartE2EDuration="2.615286164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.611947739 +0000 UTC m=+2844.159638236" watchObservedRunningTime="2026-10-30 01:30:02.615286164 +0000 UTC m=+2844.162976660"
Oct 30 01:30:02.654685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.654619    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" podStartSLOduration=2.65460081 podStartE2EDuration="2.65460081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.651626867 +0000 UTC m=+2844.199317366" watchObservedRunningTime="2026-10-30 01:30:02.65460081 +0000 UTC m=+2844.202291306"
Oct 30 01:30:02.697556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:02.697503    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" podStartSLOduration=2.697488388 podStartE2EDuration="2.697488388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.692657102 +0000 UTC m=+2844.240347618" watchObservedRunningTime="2026-10-30 01:30:02.697488388 +0000 UTC m=+2844.245178883"
Oct 30 01:30:03.328281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.328226    5104 generic.go:356] "Generic (PLEG): container finished" podID="73745591-3ee2-4e22-b43e-453ce48e765a" containerID="ea28945302a8c9e6c942c76b89bebd60d0fd51643ab13841fe9cb4978bb2b31b" exitCode=60
Oct 30 01:30:03.328834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.328328    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" event={"ID":"73745591-3ee2-4e22-b43e-453ce48e765a","Type":"ContainerDied","Data":"ea28945302a8c9e6c942c76b89bebd60d0fd51643ab13841fe9cb4978bb2b31b"}
Oct 30 01:30:03.328834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.328749    5104 scope.go:122] "RemoveContainer" containerID="ea28945302a8c9e6c942c76b89bebd60d0fd51643ab13841fe9cb4978bb2b31b"
Oct 30 01:30:03.330149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.330124    5104 generic.go:356] "Generic (PLEG): container finished" podID="b684507c-883f-403e-9090-16053c2b83fe" containerID="4242b41dc7af0f890daa41c6bf4f6efc74f90567179a2f36ddf35ee21ff72a02" exitCode=60
Oct 30 01:30:03.330292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.330220    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" event={"ID":"b684507c-883f-403e-9090-16053c2b83fe","Type":"ContainerDied","Data":"4242b41dc7af0f890daa41c6bf4f6efc74f90567179a2f36ddf35ee21ff72a02"}
Oct 30 01:30:03.330562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.330536    5104 scope.go:122] "RemoveContainer" containerID="4242b41dc7af0f890daa41c6bf4f6efc74f90567179a2f36ddf35ee21ff72a02"
Oct 30 01:30:03.332345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.332247    5104 generic.go:356] "Generic (PLEG): container finished" podID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerID="bf045b0029c4e472fc7329c062e374ce41f4e029b48f995b27cf2ff473ff1559" exitCode=60
Oct 30 01:30:03.332345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.332324    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" event={"ID":"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840","Type":"ContainerDied","Data":"bf045b0029c4e472fc7329c062e374ce41f4e029b48f995b27cf2ff473ff1559"}
Oct 30 01:30:03.332716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.332640    5104 scope.go:122] "RemoveContainer" containerID="bf045b0029c4e472fc7329c062e374ce41f4e029b48f995b27cf2ff473ff1559"
Oct 30 01:30:03.334354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.334266    5104 generic.go:356] "Generic (PLEG): container finished" podID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerID="7ae9ffd1ea652791b3c1d609c9b12a7b1d0205ae6678da406f5f992033e94414" exitCode=60
Oct 30 01:30:03.334354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.334341    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" event={"ID":"42a0b7f7-8c25-4635-8da5-efba3220e87a","Type":"ContainerDied","Data":"7ae9ffd1ea652791b3c1d609c9b12a7b1d0205ae6678da406f5f992033e94414"}
Oct 30 01:30:03.334564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.334546    5104 scope.go:122] "RemoveContainer" containerID="7ae9ffd1ea652791b3c1d609c9b12a7b1d0205ae6678da406f5f992033e94414"
Oct 30 01:30:03.336023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.336001    5104 generic.go:356] "Generic (PLEG): container finished" podID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerID="2b1186b6aca36a30a367ab3e4d38644be0d1a254b1cb1b7b8d8e90cb4b1c9199" exitCode=60
Oct 30 01:30:03.336138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.336075    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerDied","Data":"2b1186b6aca36a30a367ab3e4d38644be0d1a254b1cb1b7b8d8e90cb4b1c9199"}
Oct 30 01:30:03.336330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.336314    5104 scope.go:122] "RemoveContainer" containerID="2b1186b6aca36a30a367ab3e4d38644be0d1a254b1cb1b7b8d8e90cb4b1c9199"
Oct 30 01:30:03.338018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.337996    5104 generic.go:356] "Generic (PLEG): container finished" podID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerID="a357b1f4f7144b64965322472f770354d302ef3c6daf2f502802f71bd92412e9" exitCode=60
Oct 30 01:30:03.338120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.338067    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" event={"ID":"96a1808d-54fb-4bdc-865b-e293422c0d29","Type":"ContainerDied","Data":"a357b1f4f7144b64965322472f770354d302ef3c6daf2f502802f71bd92412e9"}
Oct 30 01:30:03.338120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.338104    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" event={"ID":"96a1808d-54fb-4bdc-865b-e293422c0d29","Type":"ContainerStarted","Data":"8fc20e7aaccafd5ba77c9ecde128169249d6c5776b915a16f9f18f4175a8cc9b"}
Oct 30 01:30:03.339822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.339803    5104 generic.go:356] "Generic (PLEG): container finished" podID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerID="dabcd91e0df518c403b9af2585849b7560c71c39e930dbd0b2eefb71c4b3cc11" exitCode=60
Oct 30 01:30:03.339912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.339882    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" event={"ID":"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c","Type":"ContainerDied","Data":"dabcd91e0df518c403b9af2585849b7560c71c39e930dbd0b2eefb71c4b3cc11"}
Oct 30 01:30:03.339974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.339932    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" event={"ID":"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c","Type":"ContainerStarted","Data":"9eb99e40e2f4a403a3a2bb3b3c25281e73de109889a34284a2b46bde5167fe94"}
Oct 30 01:30:03.341434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.341414    5104 generic.go:356] "Generic (PLEG): container finished" podID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerID="2c1c652c55c1afa8bc80091ad02b482b466e39e384a471ce72448a1e730295c7" exitCode=60
Oct 30 01:30:03.341578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.341494    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerDied","Data":"2c1c652c55c1afa8bc80091ad02b482b466e39e384a471ce72448a1e730295c7"}
Oct 30 01:30:03.341762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.341742    5104 scope.go:122] "RemoveContainer" containerID="2c1c652c55c1afa8bc80091ad02b482b466e39e384a471ce72448a1e730295c7"
Oct 30 01:30:03.343446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.343409    5104 generic.go:356] "Generic (PLEG): container finished" podID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerID="9f5ce9e9493a59177200da885f2aa70fa7c1b8e4b25707a6aaa78f239df06327" exitCode=60
Oct 30 01:30:03.343578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.343450    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" event={"ID":"6f5e8754-fd77-46a6-b7a6-d933757dc0a2","Type":"ContainerDied","Data":"9f5ce9e9493a59177200da885f2aa70fa7c1b8e4b25707a6aaa78f239df06327"}
Oct 30 01:30:03.343997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.343974    5104 scope.go:122] "RemoveContainer" containerID="9f5ce9e9493a59177200da885f2aa70fa7c1b8e4b25707a6aaa78f239df06327"
Oct 30 01:30:03.422401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.421695    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" podStartSLOduration=3.421655736 podStartE2EDuration="3.421655736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:03.420051614 +0000 UTC m=+2844.967742111" watchObservedRunningTime="2026-10-30 01:30:03.421655736 +0000 UTC m=+2844.969346234"
Oct 30 01:30:03.686809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:03.686755    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" podStartSLOduration=3.686739085 podStartE2EDuration="3.686739085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:03.683117261 +0000 UTC m=+2845.230807757" watchObservedRunningTime="2026-10-30 01:30:03.686739085 +0000 UTC m=+2845.234429581"
Oct 30 01:30:04.225601 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.225563    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"]
Oct 30 01:30:04.252377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.252300    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"]
Oct 30 01:30:04.252839 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.252738    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.280596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.279841    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-679bd2-dockercfg-flq67\""
Oct 30 01:30:04.280596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.279999    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-679bd2\""
Oct 30 01:30:04.327575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.325122    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ad5ff-2880-4e4f-bc12-5690e75e570a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.327575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.325206    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlrp\" (UniqueName: \"kubernetes.io/projected/a18ad5ff-2880-4e4f-bc12-5690e75e570a-kube-api-access-jdlrp\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.343476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.343424    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"]
Oct 30 01:30:04.351056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.351026    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.363750 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.363717    5104 generic.go:356] "Generic (PLEG): container finished" podID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerID="8fc20e7aaccafd5ba77c9ecde128169249d6c5776b915a16f9f18f4175a8cc9b" exitCode=60
Oct 30 01:30:04.363920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.363853    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" event={"ID":"96a1808d-54fb-4bdc-865b-e293422c0d29","Type":"ContainerDied","Data":"8fc20e7aaccafd5ba77c9ecde128169249d6c5776b915a16f9f18f4175a8cc9b"}
Oct 30 01:30:04.363981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.363920    5104 scope.go:122] "RemoveContainer" containerID="a357b1f4f7144b64965322472f770354d302ef3c6daf2f502802f71bd92412e9"
Oct 30 01:30:04.364216 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.364193    5104 scope.go:122] "RemoveContainer" containerID="8fc20e7aaccafd5ba77c9ecde128169249d6c5776b915a16f9f18f4175a8cc9b"
Oct 30 01:30:04.364742 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:04.364714    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7_odh-ai-gateway-infra(96a1808d-54fb-4bdc-865b-e293422c0d29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29"
Oct 30 01:30:04.381556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.376727    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"]
Oct 30 01:30:04.395766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.395731    5104 generic.go:356] "Generic (PLEG): container finished" podID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerID="9eb99e40e2f4a403a3a2bb3b3c25281e73de109889a34284a2b46bde5167fe94" exitCode=60
Oct 30 01:30:04.396114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.396078    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" event={"ID":"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c","Type":"ContainerDied","Data":"9eb99e40e2f4a403a3a2bb3b3c25281e73de109889a34284a2b46bde5167fe94"}
Oct 30 01:30:04.402867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.402836    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerStarted","Data":"2f4f0d73b241bd8d54eff74fb4af4701032b558da81214855a680fa0c7b3905b"}
Oct 30 01:30:04.403054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.396814    5104 scope.go:122] "RemoveContainer" containerID="9eb99e40e2f4a403a3a2bb3b3c25281e73de109889a34284a2b46bde5167fe94"
Oct 30 01:30:04.403549 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:04.403524    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l_odh-ai-gateway-infra(e51b9f4c-2235-4095-860f-c0e5f8b7cd8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c"
Oct 30 01:30:04.403685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.403637    5104 scope.go:122] "RemoveContainer" containerID="dabcd91e0df518c403b9af2585849b7560c71c39e930dbd0b2eefb71c4b3cc11"
Oct 30 01:30:04.428087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.428043    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8fe731-d987-4488-9c4e-ff7a5090d781-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.431520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.428433    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ad5ff-2880-4e4f-bc12-5690e75e570a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.431520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.430984    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlrp\" (UniqueName: \"kubernetes.io/projected/a18ad5ff-2880-4e4f-bc12-5690e75e570a-kube-api-access-jdlrp\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.431520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.431085    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46hb\" (UniqueName: \"kubernetes.io/projected/9a8fe731-d987-4488-9c4e-ff7a5090d781-kube-api-access-l46hb\") pod \"payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.431986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.429625    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" event={"ID":"6f5e8754-fd77-46a6-b7a6-d933757dc0a2","Type":"ContainerStarted","Data":"c4154b8814ad4d0eede12ac1a4e20a73b0505efdf94a183f4e2e35c4e96cf531"}
Oct 30 01:30:04.432926 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.430342    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ad5ff-2880-4e4f-bc12-5690e75e570a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.467366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.467326    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" event={"ID":"73745591-3ee2-4e22-b43e-453ce48e765a","Type":"ContainerStarted","Data":"a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607"}
Oct 30 01:30:04.468305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.468276    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" event={"ID":"b684507c-883f-403e-9090-16053c2b83fe","Type":"ContainerStarted","Data":"ef245fb0e71ec79f7a59748c95cc30ec790de79e1a465be35797a6ed2e13cd1a"}
Oct 30 01:30:04.468559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.468537    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlrp\" (UniqueName: \"kubernetes.io/projected/a18ad5ff-2880-4e4f-bc12-5690e75e570a-kube-api-access-jdlrp\") pod \"payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.470976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.470957    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" event={"ID":"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840","Type":"ContainerStarted","Data":"b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1"}
Oct 30 01:30:04.476520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.473833    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" event={"ID":"42a0b7f7-8c25-4635-8da5-efba3220e87a","Type":"ContainerStarted","Data":"5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf"}
Oct 30 01:30:04.478054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.477076    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerStarted","Data":"8f489da89751898d02c932f9ece8170799d19449dba03224519826edb906d78f"}
Oct 30 01:30:04.532346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.532232    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8fe731-d987-4488-9c4e-ff7a5090d781-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.532669 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.532615    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46hb\" (UniqueName: \"kubernetes.io/projected/9a8fe731-d987-4488-9c4e-ff7a5090d781-kube-api-access-l46hb\") pod \"payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.533296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.533247    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8fe731-d987-4488-9c4e-ff7a5090d781-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.557527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.557490    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46hb\" (UniqueName: \"kubernetes.io/projected/9a8fe731-d987-4488-9c4e-ff7a5090d781-kube-api-access-l46hb\") pod \"payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.621703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.621667    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:04.686301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.686072    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:04.816558 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:04.816428    5104 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0b81d4_dd2c_4140_8ad3_e13a4da9d840.slice/crio-conmon-b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0b81d4_dd2c_4140_8ad3_e13a4da9d840.slice/crio-b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a0b7f7_8c25_4635_8da5_efba3220e87a.slice/crio-5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73745591_3ee2_4e22_b43e_453ce48e765a.slice/crio-a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:30:04.936137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.936092    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main" containerID="cri-o://0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb" gracePeriod=45
Oct 30 01:30:04.951335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.946581    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main" containerID="cri-o://bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a" gracePeriod=45
Oct 30 01:30:04.954354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:04.954318    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"]
Oct 30 01:30:04.984152 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:04.984116    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18ad5ff_2880_4e4f_bc12_5690e75e570a.slice/crio-a6ed0ceefd6bd711df16df35ac6ba2f8ae03d8f8742e605cb783c10b8426f68b WatchSource:0}: Error finding container a6ed0ceefd6bd711df16df35ac6ba2f8ae03d8f8742e605cb783c10b8426f68b: Status 404 returned error can't find the container with id a6ed0ceefd6bd711df16df35ac6ba2f8ae03d8f8742e605cb783c10b8426f68b
Oct 30 01:30:05.114196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.114161    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"]
Oct 30 01:30:05.250190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.249910    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:30:05.255826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.255780    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d558cb04-b45c-478a-aa73-b0862f1d98c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d558cb04-b45c-478a-aa73-b0862f1d98c9-tls-certs\") pod \"d558cb04-b45c-478a-aa73-b0862f1d98c9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") "
Oct 30 01:30:05.256486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.256427    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-tmp-dir\") pod \"d558cb04-b45c-478a-aa73-b0862f1d98c9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") "
Oct 30 01:30:05.256808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.256735    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d558cb04-b45c-478a-aa73-b0862f1d98c9-kube-api-access-jjd7l\" (UniqueName: \"kubernetes.io/projected/d558cb04-b45c-478a-aa73-b0862f1d98c9-kube-api-access-jjd7l\") pod \"d558cb04-b45c-478a-aa73-b0862f1d98c9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") "
Oct 30 01:30:05.257009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.256992    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-home\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-home\") pod \"d558cb04-b45c-478a-aa73-b0862f1d98c9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") "
Oct 30 01:30:05.257245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.257211    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-model-cache\") pod \"d558cb04-b45c-478a-aa73-b0862f1d98c9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") "
Oct 30 01:30:05.257494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.257474    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-dshm\") pod \"d558cb04-b45c-478a-aa73-b0862f1d98c9\" (UID: \"d558cb04-b45c-478a-aa73-b0862f1d98c9\") "
Oct 30 01:30:05.266975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.266913    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-tmp-dir" pod "d558cb04-b45c-478a-aa73-b0862f1d98c9" (UID: "d558cb04-b45c-478a-aa73-b0862f1d98c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.267137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.267034    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d558cb04-b45c-478a-aa73-b0862f1d98c9-tls-certs" pod "d558cb04-b45c-478a-aa73-b0862f1d98c9" (UID: "d558cb04-b45c-478a-aa73-b0862f1d98c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:05.267331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.267299    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-home" pod "d558cb04-b45c-478a-aa73-b0862f1d98c9" (UID: "d558cb04-b45c-478a-aa73-b0862f1d98c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.303050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.302940    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-model-cache" pod "d558cb04-b45c-478a-aa73-b0862f1d98c9" (UID: "d558cb04-b45c-478a-aa73-b0862f1d98c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.304165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.304115    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-dshm" pod "d558cb04-b45c-478a-aa73-b0862f1d98c9" (UID: "d558cb04-b45c-478a-aa73-b0862f1d98c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.304587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.304546    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d558cb04-b45c-478a-aa73-b0862f1d98c9-kube-api-access-jjd7l" pod "d558cb04-b45c-478a-aa73-b0862f1d98c9" (UID: "d558cb04-b45c-478a-aa73-b0862f1d98c9"). InnerVolumeSpecName "kube-api-access-jjd7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:05.339513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.339481    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:30:05.359442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.359403    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-tmp-dir\") pod \"2a302533-defe-4800-ae63-74070ee988a5\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") "
Oct 30 01:30:05.360105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.360081    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-home\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-home\") pod \"2a302533-defe-4800-ae63-74070ee988a5\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") "
Oct 30 01:30:05.360358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.359833    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-tmp-dir" pod "2a302533-defe-4800-ae63-74070ee988a5" (UID: "2a302533-defe-4800-ae63-74070ee988a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.360506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.360389    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-model-cache\") pod \"2a302533-defe-4800-ae63-74070ee988a5\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") "
Oct 30 01:30:05.360506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.360428    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-home" pod "2a302533-defe-4800-ae63-74070ee988a5" (UID: "2a302533-defe-4800-ae63-74070ee988a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.360735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.360446    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a302533-defe-4800-ae63-74070ee988a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a302533-defe-4800-ae63-74070ee988a5-tls-certs\") pod \"2a302533-defe-4800-ae63-74070ee988a5\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") "
Oct 30 01:30:05.360829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.360776    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a302533-defe-4800-ae63-74070ee988a5-kube-api-access-wrz2n\" (UniqueName: \"kubernetes.io/projected/2a302533-defe-4800-ae63-74070ee988a5-kube-api-access-wrz2n\") pod \"2a302533-defe-4800-ae63-74070ee988a5\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") "
Oct 30 01:30:05.360916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.360637    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-model-cache" pod "2a302533-defe-4800-ae63-74070ee988a5" (UID: "2a302533-defe-4800-ae63-74070ee988a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.361313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.361269    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-dshm\") pod \"2a302533-defe-4800-ae63-74070ee988a5\" (UID: \"2a302533-defe-4800-ae63-74070ee988a5\") "
Oct 30 01:30:05.362254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362229    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362259    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d558cb04-b45c-478a-aa73-b0862f1d98c9-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362298    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362314    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362327    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362341    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjd7l\" (UniqueName: \"kubernetes.io/projected/d558cb04-b45c-478a-aa73-b0862f1d98c9-kube-api-access-jjd7l\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362355    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362370    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.362381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.362384    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d558cb04-b45c-478a-aa73-b0862f1d98c9-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.366777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.366750    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-dshm" pod "2a302533-defe-4800-ae63-74070ee988a5" (UID: "2a302533-defe-4800-ae63-74070ee988a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.370399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.370351    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a302533-defe-4800-ae63-74070ee988a5-tls-certs" pod "2a302533-defe-4800-ae63-74070ee988a5" (UID: "2a302533-defe-4800-ae63-74070ee988a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:05.371881 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.371860    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a302533-defe-4800-ae63-74070ee988a5-kube-api-access-wrz2n" pod "2a302533-defe-4800-ae63-74070ee988a5" (UID: "2a302533-defe-4800-ae63-74070ee988a5"). InnerVolumeSpecName "kube-api-access-wrz2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:05.464679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.464596    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a302533-defe-4800-ae63-74070ee988a5-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.464679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.464636    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrz2n\" (UniqueName: \"kubernetes.io/projected/2a302533-defe-4800-ae63-74070ee988a5-kube-api-access-wrz2n\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.464679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.464652    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a302533-defe-4800-ae63-74070ee988a5-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.516304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.515763    5104 generic.go:356] "Generic (PLEG): container finished" podID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerID="c4154b8814ad4d0eede12ac1a4e20a73b0505efdf94a183f4e2e35c4e96cf531" exitCode=60
Oct 30 01:30:05.516304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.515877    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" event={"ID":"6f5e8754-fd77-46a6-b7a6-d933757dc0a2","Type":"ContainerDied","Data":"c4154b8814ad4d0eede12ac1a4e20a73b0505efdf94a183f4e2e35c4e96cf531"}
Oct 30 01:30:05.516304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.515972    5104 scope.go:122] "RemoveContainer" containerID="9f5ce9e9493a59177200da885f2aa70fa7c1b8e4b25707a6aaa78f239df06327"
Oct 30 01:30:05.517144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.516714    5104 scope.go:122] "RemoveContainer" containerID="c4154b8814ad4d0eede12ac1a4e20a73b0505efdf94a183f4e2e35c4e96cf531"
Oct 30 01:30:05.517277 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.517250    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v_odh-ai-gateway-infra(6f5e8754-fd77-46a6-b7a6-d933757dc0a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2"
Oct 30 01:30:05.521403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.521382    5104 generic.go:356] "Generic (PLEG): container finished" podID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerID="dd393ab5ad3833c43e68ca0acd607b8c40768b391c464436cdde00e64441b09e" exitCode=0
Oct 30 01:30:05.521756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.521525    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" event={"ID":"c6b4e2db-b55a-428a-ae08-44d6378ccbf7","Type":"ContainerDied","Data":"dd393ab5ad3833c43e68ca0acd607b8c40768b391c464436cdde00e64441b09e"}
Oct 30 01:30:05.524686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.524624    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm" event={"ID":"9a8fe731-d987-4488-9c4e-ff7a5090d781","Type":"ContainerStarted","Data":"f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef"}
Oct 30 01:30:05.524896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.524838    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm" event={"ID":"9a8fe731-d987-4488-9c4e-ff7a5090d781","Type":"ContainerStarted","Data":"27e4de0af9fb917e75dc1c050a52c524717ee8234c39e5505db772bb8673e99e"}
Oct 30 01:30:05.526372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.526322    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:05.526985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.526962    5104 generic.go:356] "Generic (PLEG): container finished" podID="73745591-3ee2-4e22-b43e-453ce48e765a" containerID="a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607" exitCode=60
Oct 30 01:30:05.527666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.527109    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" event={"ID":"73745591-3ee2-4e22-b43e-453ce48e765a","Type":"ContainerDied","Data":"a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607"}
Oct 30 01:30:05.527822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.527512    5104 scope.go:122] "RemoveContainer" containerID="a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607"
Oct 30 01:30:05.528320 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.528298    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp_odh-ai-gateway-infra(73745591-3ee2-4e22-b43e-453ce48e765a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" podUID="73745591-3ee2-4e22-b43e-453ce48e765a"
Oct 30 01:30:05.538406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.530874    5104 generic.go:356] "Generic (PLEG): container finished" podID="b684507c-883f-403e-9090-16053c2b83fe" containerID="ef245fb0e71ec79f7a59748c95cc30ec790de79e1a465be35797a6ed2e13cd1a" exitCode=60
Oct 30 01:30:05.538406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.530891    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" event={"ID":"b684507c-883f-403e-9090-16053c2b83fe","Type":"ContainerDied","Data":"ef245fb0e71ec79f7a59748c95cc30ec790de79e1a465be35797a6ed2e13cd1a"}
Oct 30 01:30:05.538406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.531240    5104 scope.go:122] "RemoveContainer" containerID="ef245fb0e71ec79f7a59748c95cc30ec790de79e1a465be35797a6ed2e13cd1a"
Oct 30 01:30:05.538406 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.534180    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq_odh-ai-gateway-infra(b684507c-883f-403e-9090-16053c2b83fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" podUID="b684507c-883f-403e-9090-16053c2b83fe"
Oct 30 01:30:05.538406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.535015    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" containerID="cri-o://c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968" gracePeriod=45
Oct 30 01:30:05.543801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.540699    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794" event={"ID":"a18ad5ff-2880-4e4f-bc12-5690e75e570a","Type":"ContainerStarted","Data":"2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62"}
Oct 30 01:30:05.543801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.540748    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794" event={"ID":"a18ad5ff-2880-4e4f-bc12-5690e75e570a","Type":"ContainerStarted","Data":"a6ed0ceefd6bd711df16df35ac6ba2f8ae03d8f8742e605cb783c10b8426f68b"}
Oct 30 01:30:05.543801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.541399    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:05.548584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.548410    5104 generic.go:356] "Generic (PLEG): container finished" podID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerID="bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a" exitCode=0
Oct 30 01:30:05.548584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.548517    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" event={"ID":"d558cb04-b45c-478a-aa73-b0862f1d98c9","Type":"ContainerDied","Data":"bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a"}
Oct 30 01:30:05.548584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.548559    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9" event={"ID":"d558cb04-b45c-478a-aa73-b0862f1d98c9","Type":"ContainerDied","Data":"51d1fe0f2211819080a3a5e1e84783aef578bcc37f8016c391eabf76255408ed"}
Oct 30 01:30:05.548787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.548660    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"
Oct 30 01:30:05.552038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.551854    5104 scope.go:122] "RemoveContainer" containerID="ea28945302a8c9e6c942c76b89bebd60d0fd51643ab13841fe9cb4978bb2b31b"
Oct 30 01:30:05.557243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.557218    5104 generic.go:356] "Generic (PLEG): container finished" podID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerID="b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1" exitCode=60
Oct 30 01:30:05.562576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.557505    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" event={"ID":"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840","Type":"ContainerDied","Data":"b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1"}
Oct 30 01:30:05.562809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.558106    5104 scope.go:122] "RemoveContainer" containerID="b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1"
Oct 30 01:30:05.578436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.577374    5104 generic.go:356] "Generic (PLEG): container finished" podID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerID="5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf" exitCode=60
Oct 30 01:30:05.578436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.577539    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" event={"ID":"42a0b7f7-8c25-4635-8da5-efba3220e87a","Type":"ContainerDied","Data":"5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf"}
Oct 30 01:30:05.578436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.577925    5104 scope.go:122] "RemoveContainer" containerID="5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf"
Oct 30 01:30:05.578436 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.578201    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn_odh-ai-gateway-infra(42a0b7f7-8c25-4635-8da5-efba3220e87a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a"
Oct 30 01:30:05.585433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.581831    5104 generic.go:356] "Generic (PLEG): container finished" podID="2a302533-defe-4800-ae63-74070ee988a5" containerID="0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb" exitCode=0
Oct 30 01:30:05.585433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.581923    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" event={"ID":"2a302533-defe-4800-ae63-74070ee988a5","Type":"ContainerDied","Data":"0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb"}
Oct 30 01:30:05.585433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.581954    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv" event={"ID":"2a302533-defe-4800-ae63-74070ee988a5","Type":"ContainerDied","Data":"5e817f3d76ab8a92041fd1e8aae5663802cabcb61698a1a868a30420b0e07700"}
Oct 30 01:30:05.585433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.582024    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"
Oct 30 01:30:05.590378 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.589607    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f_odh-ai-gateway-infra(fb0b81d4-dd2c-4140-8ad3-e13a4da9d840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840"
Oct 30 01:30:05.598653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.597015    5104 generic.go:356] "Generic (PLEG): container finished" podID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerID="8f489da89751898d02c932f9ece8170799d19449dba03224519826edb906d78f" exitCode=60
Oct 30 01:30:05.598653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.597219    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerDied","Data":"8f489da89751898d02c932f9ece8170799d19449dba03224519826edb906d78f"}
Oct 30 01:30:05.598653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.598131    5104 scope.go:122] "RemoveContainer" containerID="8f489da89751898d02c932f9ece8170799d19449dba03224519826edb906d78f"
Oct 30 01:30:05.598653 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.598579    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9_odh-ai-gateway-infra(079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8"
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.602896    5104 scope.go:122] "RemoveContainer" containerID="8fc20e7aaccafd5ba77c9ecde128169249d6c5776b915a16f9f18f4175a8cc9b"
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.603250    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7_odh-ai-gateway-infra(96a1808d-54fb-4bdc-865b-e293422c0d29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29"
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.605666    5104 scope.go:122] "RemoveContainer" containerID="9eb99e40e2f4a403a3a2bb3b3c25281e73de109889a34284a2b46bde5167fe94"
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.605968    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l_odh-ai-gateway-infra(e51b9f4c-2235-4095-860f-c0e5f8b7cd8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c"
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.608348    5104 generic.go:356] "Generic (PLEG): container finished" podID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerID="2f4f0d73b241bd8d54eff74fb4af4701032b558da81214855a680fa0c7b3905b" exitCode=60
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.608393    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerDied","Data":"2f4f0d73b241bd8d54eff74fb4af4701032b558da81214855a680fa0c7b3905b"}
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.608746    5104 scope.go:122] "RemoveContainer" containerID="2f4f0d73b241bd8d54eff74fb4af4701032b558da81214855a680fa0c7b3905b"
Oct 30 01:30:05.609096 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.609014    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw_odh-ai-gateway-infra(1969249d-ee44-4685-86de-2a77d4f02ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8"
Oct 30 01:30:05.646784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.646753    5104 scope.go:122] "RemoveContainer" containerID="4242b41dc7af0f890daa41c6bf4f6efc74f90567179a2f36ddf35ee21ff72a02"
Oct 30 01:30:05.701844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.701634    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm" podStartSLOduration=1.701614953 podStartE2EDuration="1.701614953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:05.700219832 +0000 UTC m=+2847.247910328" watchObservedRunningTime="2026-10-30 01:30:05.701614953 +0000 UTC m=+2847.249305450"
Oct 30 01:30:05.727280 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.727247    5104 scope.go:122] "RemoveContainer" containerID="bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a"
Oct 30 01:30:05.730396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.730282    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:30:05.748522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.748448    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794" podStartSLOduration=1.748425363 podStartE2EDuration="1.748425363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:05.7423068 +0000 UTC m=+2847.289997296" watchObservedRunningTime="2026-10-30 01:30:05.748425363 +0000 UTC m=+2847.296115864"
Oct 30 01:30:05.749330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.749213    5104 scope.go:122] "RemoveContainer" containerID="bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a"
Oct 30 01:30:05.749980 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.749806    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a\": container with ID starting with bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a not found: ID does not exist" containerID="bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a"
Oct 30 01:30:05.749980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.749842    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a"} err="failed to get container status \"bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a\": rpc error: code = NotFound desc = could not find container \"bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a\": container with ID starting with bbf927e4158ec32e144c6f991cc86a54067020a16e2d7f16eb105649be8d569a not found: ID does not exist"
Oct 30 01:30:05.749980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.749882    5104 scope.go:122] "RemoveContainer" containerID="bf045b0029c4e472fc7329c062e374ce41f4e029b48f995b27cf2ff473ff1559"
Oct 30 01:30:05.757934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.757897    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 01:30:05.767102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.766757    5104 scope.go:122] "RemoveContainer" containerID="7ae9ffd1ea652791b3c1d609c9b12a7b1d0205ae6678da406f5f992033e94414"
Oct 30 01:30:05.768253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.768231    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-plugins-config-volume\") pod \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") "
Oct 30 01:30:05.768348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.768297    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-kube-api-access-l9nml\" (UniqueName: \"kubernetes.io/projected/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-kube-api-access-l9nml\") pod \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\" (UID: \"c6b4e2db-b55a-428a-ae08-44d6378ccbf7\") "
Oct 30 01:30:05.769172 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.769126    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-plugins-config-volume" pod "c6b4e2db-b55a-428a-ae08-44d6378ccbf7" (UID: "c6b4e2db-b55a-428a-ae08-44d6378ccbf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:05.772341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.771085    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-kube-api-access-l9nml" pod "c6b4e2db-b55a-428a-ae08-44d6378ccbf7" (UID: "c6b4e2db-b55a-428a-ae08-44d6378ccbf7"). InnerVolumeSpecName "kube-api-access-l9nml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:05.785123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.785077    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"]
Oct 30 01:30:05.824680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.824650    5104 scope.go:122] "RemoveContainer" containerID="0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb"
Oct 30 01:30:05.828682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.828649    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/simulator-w4-d827b7-kserve-844876ff99-2fdw9"]
Oct 30 01:30:05.860537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.860515    5104 scope.go:122] "RemoveContainer" containerID="0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb"
Oct 30 01:30:05.861141 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:05.861005    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb\": container with ID starting with 0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb not found: ID does not exist" containerID="0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb"
Oct 30 01:30:05.861141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.861045    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb"} err="failed to get container status \"0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb\": rpc error: code = NotFound desc = could not find container \"0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb\": container with ID starting with 0c6411177c61589dcfe5169858312a92e2d656afc338da7487e65ead9771a7bb not found: ID does not exist"
Oct 30 01:30:05.861141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.861072    5104 scope.go:122] "RemoveContainer" containerID="2b1186b6aca36a30a367ab3e4d38644be0d1a254b1cb1b7b8d8e90cb4b1c9199"
Oct 30 01:30:05.869815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.869773    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9nml\" (UniqueName: \"kubernetes.io/projected/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-kube-api-access-l9nml\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.869815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.869797    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4e2db-b55a-428a-ae08-44d6378ccbf7-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:05.885059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.885032    5104 scope.go:122] "RemoveContainer" containerID="2c1c652c55c1afa8bc80091ad02b482b466e39e384a471ce72448a1e730295c7"
Oct 30 01:30:05.932934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.932902    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:30:05.971261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.971224    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-model-cache\") pod \"f7aad504-0d68-4aa8-897a-6f443646e3c3\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") "
Oct 30 01:30:05.971472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.971286    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-tmp-dir\") pod \"f7aad504-0d68-4aa8-897a-6f443646e3c3\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") "
Oct 30 01:30:05.971472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.971321    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-home\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-home\") pod \"f7aad504-0d68-4aa8-897a-6f443646e3c3\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") "
Oct 30 01:30:05.971472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.971353    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-dshm\") pod \"f7aad504-0d68-4aa8-897a-6f443646e3c3\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") "
Oct 30 01:30:05.971472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.971403    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7aad504-0d68-4aa8-897a-6f443646e3c3-kube-api-access-wph2s\" (UniqueName: \"kubernetes.io/projected/f7aad504-0d68-4aa8-897a-6f443646e3c3-kube-api-access-wph2s\") pod \"f7aad504-0d68-4aa8-897a-6f443646e3c3\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") "
Oct 30 01:30:05.971708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.971539    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7aad504-0d68-4aa8-897a-6f443646e3c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aad504-0d68-4aa8-897a-6f443646e3c3-tls-certs\") pod \"f7aad504-0d68-4aa8-897a-6f443646e3c3\" (UID: \"f7aad504-0d68-4aa8-897a-6f443646e3c3\") "
Oct 30 01:30:05.973180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.973154    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-model-cache" pod "f7aad504-0d68-4aa8-897a-6f443646e3c3" (UID: "f7aad504-0d68-4aa8-897a-6f443646e3c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.973495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.973368    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-tmp-dir" pod "f7aad504-0d68-4aa8-897a-6f443646e3c3" (UID: "f7aad504-0d68-4aa8-897a-6f443646e3c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.973610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.973418    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-home" pod "f7aad504-0d68-4aa8-897a-6f443646e3c3" (UID: "f7aad504-0d68-4aa8-897a-6f443646e3c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.974181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.974156    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-dshm" pod "f7aad504-0d68-4aa8-897a-6f443646e3c3" (UID: "f7aad504-0d68-4aa8-897a-6f443646e3c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:05.974770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.974737    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7aad504-0d68-4aa8-897a-6f443646e3c3-kube-api-access-wph2s" pod "f7aad504-0d68-4aa8-897a-6f443646e3c3" (UID: "f7aad504-0d68-4aa8-897a-6f443646e3c3"). InnerVolumeSpecName "kube-api-access-wph2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:05.975903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:05.975877    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7aad504-0d68-4aa8-897a-6f443646e3c3-tls-certs" pod "f7aad504-0d68-4aa8-897a-6f443646e3c3" (UID: "f7aad504-0d68-4aa8-897a-6f443646e3c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:06.074344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.074167    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aad504-0d68-4aa8-897a-6f443646e3c3-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:06.074344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.074207    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:06.074344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.074220    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:06.074344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.074234    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:06.074344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.074246    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aad504-0d68-4aa8-897a-6f443646e3c3-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:06.074344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.074259    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wph2s\" (UniqueName: \"kubernetes.io/projected/f7aad504-0d68-4aa8-897a-6f443646e3c3-kube-api-access-wph2s\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:06.075023 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.074998    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"]
Oct 30 01:30:06.081835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.081802    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-w4-d827b7-kserve-564ccc6cb9-48kzv"]
Oct 30 01:30:06.539361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.539321    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" podUID="7aab52d9-16c4-470f-97ac-6adbf20090ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:06.539831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.539598    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" podUID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:06.581916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.581868    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a302533-defe-4800-ae63-74070ee988a5" path="/var/lib/kubelet/pods/2a302533-defe-4800-ae63-74070ee988a5/volumes"
Oct 30 01:30:06.582606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.582581    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" path="/var/lib/kubelet/pods/d558cb04-b45c-478a-aa73-b0862f1d98c9/volumes"
Oct 30 01:30:06.621852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.621822    5104 scope.go:122] "RemoveContainer" containerID="c4154b8814ad4d0eede12ac1a4e20a73b0505efdf94a183f4e2e35c4e96cf531"
Oct 30 01:30:06.622258 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.622216    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v_odh-ai-gateway-infra(6f5e8754-fd77-46a6-b7a6-d933757dc0a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2"
Oct 30 01:30:06.623663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.623635    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"
Oct 30 01:30:06.623871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.623683    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt" event={"ID":"c6b4e2db-b55a-428a-ae08-44d6378ccbf7","Type":"ContainerDied","Data":"a377c6997b67103df5453c4a8edf68d6ffb15e4e9b33d569975fd458e4c7cfcb"}
Oct 30 01:30:06.623871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.623751    5104 scope.go:122] "RemoveContainer" containerID="dd393ab5ad3833c43e68ca0acd607b8c40768b391c464436cdde00e64441b09e"
Oct 30 01:30:06.626085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.626059    5104 generic.go:356] "Generic (PLEG): container finished" podID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerID="c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968" exitCode=0
Oct 30 01:30:06.626308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.626096    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" event={"ID":"f7aad504-0d68-4aa8-897a-6f443646e3c3","Type":"ContainerDied","Data":"c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968"}
Oct 30 01:30:06.626418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.626329    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh" event={"ID":"f7aad504-0d68-4aa8-897a-6f443646e3c3","Type":"ContainerDied","Data":"cf910493d8736b039b3f7a73181a773bebb30630f8f667849bd94f92a9811c05"}
Oct 30 01:30:06.626418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.626138    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"
Oct 30 01:30:06.629817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.629367    5104 scope.go:122] "RemoveContainer" containerID="a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607"
Oct 30 01:30:06.629817 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.629672    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp_odh-ai-gateway-infra(73745591-3ee2-4e22-b43e-453ce48e765a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" podUID="73745591-3ee2-4e22-b43e-453ce48e765a"
Oct 30 01:30:06.632123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.632101    5104 scope.go:122] "RemoveContainer" containerID="ef245fb0e71ec79f7a59748c95cc30ec790de79e1a465be35797a6ed2e13cd1a"
Oct 30 01:30:06.632416 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.632385    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq_odh-ai-gateway-infra(b684507c-883f-403e-9090-16053c2b83fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" podUID="b684507c-883f-403e-9090-16053c2b83fe"
Oct 30 01:30:06.636446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.636426    5104 scope.go:122] "RemoveContainer" containerID="b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1"
Oct 30 01:30:06.636713 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.636694    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f_odh-ai-gateway-infra(fb0b81d4-dd2c-4140-8ad3-e13a4da9d840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840"
Oct 30 01:30:06.638334 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.638312    5104 scope.go:122] "RemoveContainer" containerID="5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf"
Oct 30 01:30:06.638676 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.638639    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn_odh-ai-gateway-infra(42a0b7f7-8c25-4635-8da5-efba3220e87a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a"
Oct 30 01:30:06.638991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.638974    5104 scope.go:122] "RemoveContainer" containerID="c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968"
Oct 30 01:30:06.641329 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.641314    5104 scope.go:122] "RemoveContainer" containerID="8f489da89751898d02c932f9ece8170799d19449dba03224519826edb906d78f"
Oct 30 01:30:06.641607 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.641577    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9_odh-ai-gateway-infra(079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8"
Oct 30 01:30:06.643616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.643595    5104 scope.go:122] "RemoveContainer" containerID="2f4f0d73b241bd8d54eff74fb4af4701032b558da81214855a680fa0c7b3905b"
Oct 30 01:30:06.643825 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.643799    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw_odh-ai-gateway-infra(1969249d-ee44-4685-86de-2a77d4f02ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8"
Oct 30 01:30:06.650428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.650239    5104 scope.go:122] "RemoveContainer" containerID="c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968"
Oct 30 01:30:06.650556 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:06.650529    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968\": container with ID starting with c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968 not found: ID does not exist" containerID="c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968"
Oct 30 01:30:06.650610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.650566    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968"} err="failed to get container status \"c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968\": rpc error: code = NotFound desc = could not find container \"c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968\": container with ID starting with c6037a0e8523e7aba8f47488e294cb2ed2f27a86dbdae55fec3adc1355e79968 not found: ID does not exist"
Oct 30 01:30:06.896589 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.896476    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"]
Oct 30 01:30:06.904007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:06.903970    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-2-w1-c104b8-kserve-64f4fb9745-wx7fh"]
Oct 30 01:30:07.053473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:07.053386    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"]
Oct 30 01:30:07.063628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:07.063583    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-29145c-848dbcdcb7-6gkpt"]
Oct 30 01:30:07.974963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:07.974890    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"]
Oct 30 01:30:07.975921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:07.975840    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy" containerID="cri-o://467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2" gracePeriod=30
Oct 30 01:30:08.584325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.584284    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" path="/var/lib/kubelet/pods/c6b4e2db-b55a-428a-ae08-44d6378ccbf7/volumes"
Oct 30 01:30:08.585068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.585046    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" path="/var/lib/kubelet/pods/f7aad504-0d68-4aa8-897a-6f443646e3c3/volumes"
Oct 30 01:30:08.732046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732008    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"]
Oct 30 01:30:08.732856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732832    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerName="payload-pre-processing"
Oct 30 01:30:08.732856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732856    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerName="payload-pre-processing"
Oct 30 01:30:08.733046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732873    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main"
Oct 30 01:30:08.733046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732881    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main"
Oct 30 01:30:08.733046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732891    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main"
Oct 30 01:30:08.733046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732898    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main"
Oct 30 01:30:08.733046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732977    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main"
Oct 30 01:30:08.733046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.732986    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main"
Oct 30 01:30:08.733369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.733208    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7aad504-0d68-4aa8-897a-6f443646e3c3" containerName="main"
Oct 30 01:30:08.733369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.733227    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b4e2db-b55a-428a-ae08-44d6378ccbf7" containerName="payload-pre-processing"
Oct 30 01:30:08.733369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.733239    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d558cb04-b45c-478a-aa73-b0862f1d98c9" containerName="main"
Oct 30 01:30:08.733369 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.733249    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a302533-defe-4800-ae63-74070ee988a5" containerName="main"
Oct 30 01:30:08.738279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.738257    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.764653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.764625    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:08.791286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.791253    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"]
Oct 30 01:30:08.827508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.826667    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.827508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.826771    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.827508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.826804    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29sd\" (UniqueName: \"kubernetes.io/projected/121c1ac7-2e84-485a-883f-43db564126c2-kube-api-access-g29sd\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.927793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.927754    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.927994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.927867    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.927994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.927898    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29sd\" (UniqueName: \"kubernetes.io/projected/121c1ac7-2e84-485a-883f-43db564126c2-kube-api-access-g29sd\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.928818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.928782    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.930504 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.930477    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:08.951248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:08.951211    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29sd\" (UniqueName: \"kubernetes.io/projected/121c1ac7-2e84-485a-883f-43db564126c2-kube-api-access-g29sd\") pod \"maas-api-revoke-sub-3422aa3e-pjq9w\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:09.053692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:09.053658    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:09.115661 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:09.115613    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:30:09.384919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:09.384729    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"]
Oct 30 01:30:09.386687 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:09.386658    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121c1ac7_2e84_485a_883f_43db564126c2.slice/crio-cd6d8a33da94c44e5622c11c19a1636ab2d79801007c40d23ba7de9366c2573f WatchSource:0}: Error finding container cd6d8a33da94c44e5622c11c19a1636ab2d79801007c40d23ba7de9366c2573f: Status 404 returned error can't find the container with id cd6d8a33da94c44e5622c11c19a1636ab2d79801007c40d23ba7de9366c2573f
Oct 30 01:30:09.407779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:09.406961    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:09.702393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:09.702357    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w" event={"ID":"121c1ac7-2e84-485a-883f-43db564126c2","Type":"ContainerStarted","Data":"cd6d8a33da94c44e5622c11c19a1636ab2d79801007c40d23ba7de9366c2573f"}
Oct 30 01:30:10.413843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:10.413809    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" podUID="801aa334-19ec-4ae7-9c34-19bab145c954" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:10.720670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:10.720633    5104 generic.go:356] "Generic (PLEG): container finished" podID="121c1ac7-2e84-485a-883f-43db564126c2" containerID="0e3dba7217e5141ad91725f08e919d0dd83dd140fa8a7232f6a82958abc17c1d" exitCode=0
Oct 30 01:30:10.720873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:10.720725    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w" event={"ID":"121c1ac7-2e84-485a-883f-43db564126c2","Type":"ContainerDied","Data":"0e3dba7217e5141ad91725f08e919d0dd83dd140fa8a7232f6a82958abc17c1d"}
Oct 30 01:30:11.622452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:11.622415    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"]
Oct 30 01:30:11.634302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:11.634264    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-368bbb-2tmnb"]
Oct 30 01:30:11.941483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:11.941429    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:12.017120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.017084    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-openshift-ca\") pod \"121c1ac7-2e84-485a-883f-43db564126c2\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") "
Oct 30 01:30:12.017120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.017126    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-xks-ca\") pod \"121c1ac7-2e84-485a-883f-43db564126c2\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") "
Oct 30 01:30:12.017415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.017228    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121c1ac7-2e84-485a-883f-43db564126c2-kube-api-access-g29sd\" (UniqueName: \"kubernetes.io/projected/121c1ac7-2e84-485a-883f-43db564126c2-kube-api-access-g29sd\") pod \"121c1ac7-2e84-485a-883f-43db564126c2\" (UID: \"121c1ac7-2e84-485a-883f-43db564126c2\") "
Oct 30 01:30:12.017827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.017793    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-openshift-ca" pod "121c1ac7-2e84-485a-883f-43db564126c2" (UID: "121c1ac7-2e84-485a-883f-43db564126c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:12.019654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.019622    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121c1ac7-2e84-485a-883f-43db564126c2-kube-api-access-g29sd" pod "121c1ac7-2e84-485a-883f-43db564126c2" (UID: "121c1ac7-2e84-485a-883f-43db564126c2"). InnerVolumeSpecName "kube-api-access-g29sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:12.020036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.020014    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-xks-ca" pod "121c1ac7-2e84-485a-883f-43db564126c2" (UID: "121c1ac7-2e84-485a-883f-43db564126c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:12.120139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.120056    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:12.120139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.120142    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121c1ac7-2e84-485a-883f-43db564126c2-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:12.120411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.120174    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29sd\" (UniqueName: \"kubernetes.io/projected/121c1ac7-2e84-485a-883f-43db564126c2-kube-api-access-g29sd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:12.578806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.578768    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb" path="/var/lib/kubelet/pods/4ea31cb1-fd82-411f-ae4e-5a0b1f0fc1eb/volumes"
Oct 30 01:30:12.743938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.743901    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"
Oct 30 01:30:12.744431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.743898    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w" event={"ID":"121c1ac7-2e84-485a-883f-43db564126c2","Type":"ContainerDied","Data":"cd6d8a33da94c44e5622c11c19a1636ab2d79801007c40d23ba7de9366c2573f"}
Oct 30 01:30:12.744431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:12.744051    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6d8a33da94c44e5622c11c19a1636ab2d79801007c40d23ba7de9366c2573f"
Oct 30 01:30:13.358938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.358911    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:30:13.436089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.429395    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"]
Oct 30 01:30:13.442103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.442064    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-podinfo\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.442478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.442438    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-data\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.442676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.442658    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-envoy\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.442870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.442851    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-kube-api-access-hr8t9\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-kube-api-access-hr8t9\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.443010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.442993    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-credential-socket\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.443161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.443144    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-socket\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.443427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.443408    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istiod-ca-cert\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.443643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.443624    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-token\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-token\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.443841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.443820    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-certs\") pod \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\" (UID: \"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c\") "
Oct 30 01:30:13.450263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.444281    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-credential-socket" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:13.450263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.444601    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istiod-ca-cert" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:13.450263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.445838    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-data" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:13.450263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.446269    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-podinfo" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:30:13.450263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.447707    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-certs" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:13.450723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.450549    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-envoy" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:13.450723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.450566    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-kube-api-access-hr8t9" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "kube-api-access-hr8t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:13.451616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.451587    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-socket" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:13.452515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.452490    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-token" pod "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" (UID: "f62ca0a8-b831-4192-a5e7-2e6d4fe6340c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:13.574261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.568115    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.578494 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.574556    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.579757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.579727    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.581004 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.580976    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.581279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.581258    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.581388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.581376    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.581529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.581512    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr8t9\" (UniqueName: \"kubernetes.io/projected/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-kube-api-access-hr8t9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.581663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.581648    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.581759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.581746    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.753817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.753774    5104 generic.go:356] "Generic (PLEG): container finished" podID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerID="467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2" exitCode=0
Oct 30 01:30:13.754249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.753828    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"
Oct 30 01:30:13.754249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.753855    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" event={"ID":"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c","Type":"ContainerDied","Data":"467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2"}
Oct 30 01:30:13.754249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.753910    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs" event={"ID":"f62ca0a8-b831-4192-a5e7-2e6d4fe6340c","Type":"ContainerDied","Data":"c5ec2d8e1895da2fddf8fb59842b2277d0b400461d20703f4a9a57f7dd325ff1"}
Oct 30 01:30:13.754249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.753936    5104 scope.go:122] "RemoveContainer" containerID="467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2"
Oct 30 01:30:13.766084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.766064    5104 scope.go:122] "RemoveContainer" containerID="467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2"
Oct 30 01:30:13.766398 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:13.766375    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2\": container with ID starting with 467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2 not found: ID does not exist" containerID="467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2"
Oct 30 01:30:13.766450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.766407    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2"} err="failed to get container status \"467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2\": rpc error: code = NotFound desc = could not find container \"467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2\": container with ID starting with 467044834c708aa95855f448da3e911c3df0b503c34f41cbf47a8dae2e374cb2 not found: ID does not exist"
Oct 30 01:30:13.817034 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.816997    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"]
Oct 30 01:30:13.823961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:13.823921    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qx4vs"]
Oct 30 01:30:14.570447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:14.570411    5104 scope.go:122] "RemoveContainer" containerID="8fc20e7aaccafd5ba77c9ecde128169249d6c5776b915a16f9f18f4175a8cc9b"
Oct 30 01:30:14.580169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:14.580138    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" path="/var/lib/kubelet/pods/f62ca0a8-b831-4192-a5e7-2e6d4fe6340c/volumes"
Oct 30 01:30:15.570560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:15.570525    5104 scope.go:122] "RemoveContainer" containerID="ef245fb0e71ec79f7a59748c95cc30ec790de79e1a465be35797a6ed2e13cd1a"
Oct 30 01:30:15.570935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:15.570640    5104 scope.go:122] "RemoveContainer" containerID="9eb99e40e2f4a403a3a2bb3b3c25281e73de109889a34284a2b46bde5167fe94"
Oct 30 01:30:15.778244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:15.778162    5104 generic.go:356] "Generic (PLEG): container finished" podID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerID="55fcef6b83bc68bd11efe0e14a54cfe0f48b6785f487bbfea7114ac5fb0d2a51" exitCode=60
Oct 30 01:30:15.778404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:15.778234    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" event={"ID":"96a1808d-54fb-4bdc-865b-e293422c0d29","Type":"ContainerDied","Data":"55fcef6b83bc68bd11efe0e14a54cfe0f48b6785f487bbfea7114ac5fb0d2a51"}
Oct 30 01:30:15.778404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:15.778303    5104 scope.go:122] "RemoveContainer" containerID="8fc20e7aaccafd5ba77c9ecde128169249d6c5776b915a16f9f18f4175a8cc9b"
Oct 30 01:30:15.778885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:15.778860    5104 scope.go:122] "RemoveContainer" containerID="55fcef6b83bc68bd11efe0e14a54cfe0f48b6785f487bbfea7114ac5fb0d2a51"
Oct 30 01:30:15.779278 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:15.779256    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7_odh-ai-gateway-infra(96a1808d-54fb-4bdc-865b-e293422c0d29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29"
Oct 30 01:30:16.538665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.538619    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" podUID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:16.538665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.538661    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" podUID="7aab52d9-16c4-470f-97ac-6adbf20090ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:16.569599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.569564    5104 scope.go:122] "RemoveContainer" containerID="2f4f0d73b241bd8d54eff74fb4af4701032b558da81214855a680fa0c7b3905b"
Oct 30 01:30:16.569781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.569624    5104 scope.go:122] "RemoveContainer" containerID="c4154b8814ad4d0eede12ac1a4e20a73b0505efdf94a183f4e2e35c4e96cf531"
Oct 30 01:30:16.793564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.793519    5104 generic.go:356] "Generic (PLEG): container finished" podID="b684507c-883f-403e-9090-16053c2b83fe" containerID="f8d856f768dfa53f9754477f97d3df4572c4403e1239d01699d1e01cce72b930" exitCode=60
Oct 30 01:30:16.793936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.793573    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" event={"ID":"b684507c-883f-403e-9090-16053c2b83fe","Type":"ContainerDied","Data":"f8d856f768dfa53f9754477f97d3df4572c4403e1239d01699d1e01cce72b930"}
Oct 30 01:30:16.793936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.793640    5104 scope.go:122] "RemoveContainer" containerID="ef245fb0e71ec79f7a59748c95cc30ec790de79e1a465be35797a6ed2e13cd1a"
Oct 30 01:30:16.794409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.794102    5104 scope.go:122] "RemoveContainer" containerID="f8d856f768dfa53f9754477f97d3df4572c4403e1239d01699d1e01cce72b930"
Oct 30 01:30:16.794576 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:16.794500    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq_odh-ai-gateway-infra(b684507c-883f-403e-9090-16053c2b83fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" podUID="b684507c-883f-403e-9090-16053c2b83fe"
Oct 30 01:30:16.800260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.800228    5104 generic.go:356] "Generic (PLEG): container finished" podID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerID="a43731f4efe9978361af9f719983d31584781aaa18ecccf95d80ccaf1cb3c876" exitCode=60
Oct 30 01:30:16.800524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.800291    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" event={"ID":"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c","Type":"ContainerDied","Data":"a43731f4efe9978361af9f719983d31584781aaa18ecccf95d80ccaf1cb3c876"}
Oct 30 01:30:16.801030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.800860    5104 scope.go:122] "RemoveContainer" containerID="a43731f4efe9978361af9f719983d31584781aaa18ecccf95d80ccaf1cb3c876"
Oct 30 01:30:16.801318 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:16.801255    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l_odh-ai-gateway-infra(e51b9f4c-2235-4095-860f-c0e5f8b7cd8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c"
Oct 30 01:30:16.846979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.846959    5104 scope.go:122] "RemoveContainer" containerID="9eb99e40e2f4a403a3a2bb3b3c25281e73de109889a34284a2b46bde5167fe94"
Oct 30 01:30:16.854103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:16.854071    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"]
Oct 30 01:30:17.102321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.102294    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:17.248392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.248352    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a1808d-54fb-4bdc-865b-e293422c0d29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a1808d-54fb-4bdc-865b-e293422c0d29-maas-api-service-ca\") pod \"96a1808d-54fb-4bdc-865b-e293422c0d29\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") "
Oct 30 01:30:17.248594 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.248444    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a1808d-54fb-4bdc-865b-e293422c0d29-kube-api-access-7pdxg\" (UniqueName: \"kubernetes.io/projected/96a1808d-54fb-4bdc-865b-e293422c0d29-kube-api-access-7pdxg\") pod \"96a1808d-54fb-4bdc-865b-e293422c0d29\" (UID: \"96a1808d-54fb-4bdc-865b-e293422c0d29\") "
Oct 30 01:30:17.248759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.248736    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a1808d-54fb-4bdc-865b-e293422c0d29-maas-api-service-ca" pod "96a1808d-54fb-4bdc-865b-e293422c0d29" (UID: "96a1808d-54fb-4bdc-865b-e293422c0d29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:17.248979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.248952    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a1808d-54fb-4bdc-865b-e293422c0d29-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:17.250685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.250660    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a1808d-54fb-4bdc-865b-e293422c0d29-kube-api-access-7pdxg" pod "96a1808d-54fb-4bdc-865b-e293422c0d29" (UID: "96a1808d-54fb-4bdc-865b-e293422c0d29"). InnerVolumeSpecName "kube-api-access-7pdxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:17.350919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.350825    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdxg\" (UniqueName: \"kubernetes.io/projected/96a1808d-54fb-4bdc-865b-e293422c0d29-kube-api-access-7pdxg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:17.570062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.570025    5104 scope.go:122] "RemoveContainer" containerID="8f489da89751898d02c932f9ece8170799d19449dba03224519826edb906d78f"
Oct 30 01:30:17.570251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.570126    5104 scope.go:122] "RemoveContainer" containerID="5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf"
Oct 30 01:30:17.695139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.695108    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:30:17.697389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.697364    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:30:17.863309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.863210    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7" event={"ID":"96a1808d-54fb-4bdc-865b-e293422c0d29","Type":"ContainerDied","Data":"e358d0513b474d05bc1c7c83454f547b746242c1431821c52a60ca92468f82f2"}
Oct 30 01:30:17.863309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.863297    5104 scope.go:122] "RemoveContainer" containerID="55fcef6b83bc68bd11efe0e14a54cfe0f48b6785f487bbfea7114ac5fb0d2a51"
Oct 30 01:30:17.863939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.863400    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"
Oct 30 01:30:17.888564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.888530    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerStarted","Data":"7e88e8e9ae6827cfc1ce278690e529e55245f571302059a30a8b753583ef3e73"}
Oct 30 01:30:17.892233 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.892207    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" event={"ID":"6f5e8754-fd77-46a6-b7a6-d933757dc0a2","Type":"ContainerDied","Data":"59d6839fa337745fe64a8bde52b9ef49bdd93c12caca1b2f66ba4e5042293b52"}
Oct 30 01:30:17.893531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.891851    5104 generic.go:356] "Generic (PLEG): container finished" podID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerID="59d6839fa337745fe64a8bde52b9ef49bdd93c12caca1b2f66ba4e5042293b52" exitCode=60
Oct 30 01:30:17.895519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.894796    5104 scope.go:122] "RemoveContainer" containerID="59d6839fa337745fe64a8bde52b9ef49bdd93c12caca1b2f66ba4e5042293b52"
Oct 30 01:30:17.895519 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:17.895153    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v_odh-ai-gateway-infra(6f5e8754-fd77-46a6-b7a6-d933757dc0a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2"
Oct 30 01:30:17.909014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.908988    5104 scope.go:122] "RemoveContainer" containerID="c4154b8814ad4d0eede12ac1a4e20a73b0505efdf94a183f4e2e35c4e96cf531"
Oct 30 01:30:17.916676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.916575    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"]
Oct 30 01:30:17.931764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:17.931731    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"]
Oct 30 01:30:18.112134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.112072    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"]
Oct 30 01:30:18.119031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.118942    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d81f0a-29888730-lhls7"]
Oct 30 01:30:18.136306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.136275    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:18.151495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.151443    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:18.182763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.182724    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b684507c-883f-403e-9090-16053c2b83fe-kube-api-access-n2r85\" (UniqueName: \"kubernetes.io/projected/b684507c-883f-403e-9090-16053c2b83fe-kube-api-access-n2r85\") pod \"b684507c-883f-403e-9090-16053c2b83fe\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") "
Oct 30 01:30:18.182931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.182811    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b684507c-883f-403e-9090-16053c2b83fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b684507c-883f-403e-9090-16053c2b83fe-maas-api-service-ca\") pod \"b684507c-883f-403e-9090-16053c2b83fe\" (UID: \"b684507c-883f-403e-9090-16053c2b83fe\") "
Oct 30 01:30:18.183013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.182985    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-maas-api-service-ca\") pod \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") "
Oct 30 01:30:18.183127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.183110    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-kube-api-access-mzww7\" (UniqueName: \"kubernetes.io/projected/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-kube-api-access-mzww7\") pod \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\" (UID: \"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c\") "
Oct 30 01:30:18.184415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.183926    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-maas-api-service-ca" pod "e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" (UID: "e51b9f4c-2235-4095-860f-c0e5f8b7cd8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:18.184590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.184565    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b684507c-883f-403e-9090-16053c2b83fe-maas-api-service-ca" pod "b684507c-883f-403e-9090-16053c2b83fe" (UID: "b684507c-883f-403e-9090-16053c2b83fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:18.187526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.187494    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b684507c-883f-403e-9090-16053c2b83fe-kube-api-access-n2r85" pod "b684507c-883f-403e-9090-16053c2b83fe" (UID: "b684507c-883f-403e-9090-16053c2b83fe"). InnerVolumeSpecName "kube-api-access-n2r85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:18.187762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.187740    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.187837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.187767    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2r85\" (UniqueName: \"kubernetes.io/projected/b684507c-883f-403e-9090-16053c2b83fe-kube-api-access-n2r85\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.187837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.187784    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b684507c-883f-403e-9090-16053c2b83fe-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.276719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.276675    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-kube-api-access-mzww7" pod "e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" (UID: "e51b9f4c-2235-4095-860f-c0e5f8b7cd8c"). InnerVolumeSpecName "kube-api-access-mzww7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:18.289610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.289535    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzww7\" (UniqueName: \"kubernetes.io/projected/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c-kube-api-access-mzww7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.574281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.574249    5104 scope.go:122] "RemoveContainer" containerID="a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607"
Oct 30 01:30:18.586118 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.586087    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" path="/var/lib/kubelet/pods/96a1808d-54fb-4bdc-865b-e293422c0d29/volumes"
Oct 30 01:30:18.934617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.934526    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"
Oct 30 01:30:18.934617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.934557    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq" event={"ID":"b684507c-883f-403e-9090-16053c2b83fe","Type":"ContainerDied","Data":"ef46ed4be5e89b53f98b06f4a9069f5a1bcf875cf0819adb7f4fab95d1b16216"}
Oct 30 01:30:18.935180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.934624    5104 scope.go:122] "RemoveContainer" containerID="f8d856f768dfa53f9754477f97d3df4572c4403e1239d01699d1e01cce72b930"
Oct 30 01:30:18.936708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.936683    5104 generic.go:356] "Generic (PLEG): container finished" podID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerID="87529c37987085cc3a21bff3f288d89f523a001ed4c64020b0dc6f7c06a6f95e" exitCode=60
Oct 30 01:30:18.936826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.936781    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" event={"ID":"42a0b7f7-8c25-4635-8da5-efba3220e87a","Type":"ContainerDied","Data":"87529c37987085cc3a21bff3f288d89f523a001ed4c64020b0dc6f7c06a6f95e"}
Oct 30 01:30:18.937179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.937162    5104 scope.go:122] "RemoveContainer" containerID="87529c37987085cc3a21bff3f288d89f523a001ed4c64020b0dc6f7c06a6f95e"
Oct 30 01:30:18.937495 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:18.937443    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn_odh-ai-gateway-infra(42a0b7f7-8c25-4635-8da5-efba3220e87a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a"
Oct 30 01:30:18.939740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.939717    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerStarted","Data":"c2a26e2bef074560bd026b837363c6d034538e911ae6c6b83b73ef763a299a21"}
Oct 30 01:30:18.942177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.942162    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"
Oct 30 01:30:18.942356 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.942166    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l" event={"ID":"e51b9f4c-2235-4095-860f-c0e5f8b7cd8c","Type":"ContainerDied","Data":"7d9d4bf4dd1cbc3990730019f6b815f2d0473f640b507a4df67b2943f9827b3d"}
Oct 30 01:30:18.944157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.944137    5104 generic.go:356] "Generic (PLEG): container finished" podID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerID="7e88e8e9ae6827cfc1ce278690e529e55245f571302059a30a8b753583ef3e73" exitCode=60
Oct 30 01:30:18.944248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.944210    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerDied","Data":"7e88e8e9ae6827cfc1ce278690e529e55245f571302059a30a8b753583ef3e73"}
Oct 30 01:30:18.944590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.944572    5104 scope.go:122] "RemoveContainer" containerID="7e88e8e9ae6827cfc1ce278690e529e55245f571302059a30a8b753583ef3e73"
Oct 30 01:30:18.944857 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:18.944831    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw_odh-ai-gateway-infra(1969249d-ee44-4685-86de-2a77d4f02ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8"
Oct 30 01:30:18.959490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.959433    5104 scope.go:122] "RemoveContainer" containerID="5d6aa5f4e610e7199cb20e88886a6480632e7ed93135fa768c663db2ea8d5cdf"
Oct 30 01:30:18.966887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.966856    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"]
Oct 30 01:30:18.982115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:18.982090    5104 scope.go:122] "RemoveContainer" containerID="a43731f4efe9978361af9f719983d31584781aaa18ecccf95d80ccaf1cb3c876"
Oct 30 01:30:19.018671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.018637    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"]
Oct 30 01:30:19.045840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.045528    5104 scope.go:122] "RemoveContainer" containerID="2f4f0d73b241bd8d54eff74fb4af4701032b558da81214855a680fa0c7b3905b"
Oct 30 01:30:19.150011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.149983    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:19.174952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.174917    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"]
Oct 30 01:30:19.186151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.186061    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888730-sgq4l"]
Oct 30 01:30:19.225993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.225955    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"]
Oct 30 01:30:19.227540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.227510    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-kube-api-access-shhc7\" (UniqueName: \"kubernetes.io/projected/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-kube-api-access-shhc7\") pod \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") "
Oct 30 01:30:19.227763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.227746    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-maas-api-service-ca\") pod \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\" (UID: \"6f5e8754-fd77-46a6-b7a6-d933757dc0a2\") "
Oct 30 01:30:19.228187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.228149    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-maas-api-service-ca" pod "6f5e8754-fd77-46a6-b7a6-d933757dc0a2" (UID: "6f5e8754-fd77-46a6-b7a6-d933757dc0a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:19.229724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.229699    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-kube-api-access-shhc7" pod "6f5e8754-fd77-46a6-b7a6-d933757dc0a2" (UID: "6f5e8754-fd77-46a6-b7a6-d933757dc0a2"). InnerVolumeSpecName "kube-api-access-shhc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:19.230164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.230142    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d827b7-29888730-m96wq"]
Oct 30 01:30:19.330225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.330167    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.330626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.330588    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhc7\" (UniqueName: \"kubernetes.io/projected/6f5e8754-fd77-46a6-b7a6-d933757dc0a2-kube-api-access-shhc7\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.401149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.401107    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:19.569559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:19.569451    5104 scope.go:122] "RemoveContainer" containerID="b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1"
Oct 30 01:30:20.002205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.002167    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v" event={"ID":"6f5e8754-fd77-46a6-b7a6-d933757dc0a2","Type":"ContainerDied","Data":"2cc7f9bae5c973f17cf893abad316aaea8578a7d8692b7ec302687fb0a78ff89"}
Oct 30 01:30:20.002856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.002835    5104 scope.go:122] "RemoveContainer" containerID="59d6839fa337745fe64a8bde52b9ef49bdd93c12caca1b2f66ba4e5042293b52"
Oct 30 01:30:20.003103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.003083    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"
Oct 30 01:30:20.024239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.013813    5104 generic.go:356] "Generic (PLEG): container finished" podID="73745591-3ee2-4e22-b43e-453ce48e765a" containerID="18446c035f18afa9610f4b64b591b7291a0cf1c343343b2bf3b00d9ee3f16264" exitCode=60
Oct 30 01:30:20.024239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.013911    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" event={"ID":"73745591-3ee2-4e22-b43e-453ce48e765a","Type":"ContainerDied","Data":"18446c035f18afa9610f4b64b591b7291a0cf1c343343b2bf3b00d9ee3f16264"}
Oct 30 01:30:20.024239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.014438    5104 scope.go:122] "RemoveContainer" containerID="18446c035f18afa9610f4b64b591b7291a0cf1c343343b2bf3b00d9ee3f16264"
Oct 30 01:30:20.024239 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:20.014873    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp_odh-ai-gateway-infra(73745591-3ee2-4e22-b43e-453ce48e765a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" podUID="73745591-3ee2-4e22-b43e-453ce48e765a"
Oct 30 01:30:20.044264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.044229    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"]
Oct 30 01:30:20.049828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.049808    5104 scope.go:122] "RemoveContainer" containerID="a821f98d66792f7e1eef8ba52c10b7bfad4d79bee48016c36d9599e8410b9607"
Oct 30 01:30:20.052513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.052439    5104 generic.go:356] "Generic (PLEG): container finished" podID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerID="c2a26e2bef074560bd026b837363c6d034538e911ae6c6b83b73ef763a299a21" exitCode=60
Oct 30 01:30:20.052642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.052621    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerDied","Data":"c2a26e2bef074560bd026b837363c6d034538e911ae6c6b83b73ef763a299a21"}
Oct 30 01:30:20.053054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.053033    5104 scope.go:122] "RemoveContainer" containerID="c2a26e2bef074560bd026b837363c6d034538e911ae6c6b83b73ef763a299a21"
Oct 30 01:30:20.053413 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:20.053393    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9_odh-ai-gateway-infra(079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8"
Oct 30 01:30:20.092171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.092144    5104 scope.go:122] "RemoveContainer" containerID="8f489da89751898d02c932f9ece8170799d19449dba03224519826edb906d78f"
Oct 30 01:30:20.242995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.241587    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"]
Oct 30 01:30:20.319022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.318916    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:20.340719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.340681    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"]
Oct 30 01:30:20.350040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.350001    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888730-q4t9v"]
Oct 30 01:30:20.387834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.387791    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a0b7f7-8c25-4635-8da5-efba3220e87a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a0b7f7-8c25-4635-8da5-efba3220e87a-maas-api-service-ca\") pod \"42a0b7f7-8c25-4635-8da5-efba3220e87a\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") "
Oct 30 01:30:20.388056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.388034    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a0b7f7-8c25-4635-8da5-efba3220e87a-kube-api-access-fx9hn\" (UniqueName: \"kubernetes.io/projected/42a0b7f7-8c25-4635-8da5-efba3220e87a-kube-api-access-fx9hn\") pod \"42a0b7f7-8c25-4635-8da5-efba3220e87a\" (UID: \"42a0b7f7-8c25-4635-8da5-efba3220e87a\") "
Oct 30 01:30:20.388222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.388200    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a0b7f7-8c25-4635-8da5-efba3220e87a-maas-api-service-ca" pod "42a0b7f7-8c25-4635-8da5-efba3220e87a" (UID: "42a0b7f7-8c25-4635-8da5-efba3220e87a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.388590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.388566    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a0b7f7-8c25-4635-8da5-efba3220e87a-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.390625 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.390415    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a0b7f7-8c25-4635-8da5-efba3220e87a-kube-api-access-fx9hn" pod "42a0b7f7-8c25-4635-8da5-efba3220e87a" (UID: "42a0b7f7-8c25-4635-8da5-efba3220e87a"). InnerVolumeSpecName "kube-api-access-fx9hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:20.414746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.413938    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" podUID="801aa334-19ec-4ae7-9c34-19bab145c954" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:30:20.440550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.440518    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:20.489670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.489632    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1969249d-ee44-4685-86de-2a77d4f02ba8-kube-api-access-b75g5\" (UniqueName: \"kubernetes.io/projected/1969249d-ee44-4685-86de-2a77d4f02ba8-kube-api-access-b75g5\") pod \"1969249d-ee44-4685-86de-2a77d4f02ba8\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") "
Oct 30 01:30:20.489670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.489677    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1969249d-ee44-4685-86de-2a77d4f02ba8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1969249d-ee44-4685-86de-2a77d4f02ba8-maas-api-service-ca\") pod \"1969249d-ee44-4685-86de-2a77d4f02ba8\" (UID: \"1969249d-ee44-4685-86de-2a77d4f02ba8\") "
Oct 30 01:30:20.490132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.490107    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1969249d-ee44-4685-86de-2a77d4f02ba8-maas-api-service-ca" pod "1969249d-ee44-4685-86de-2a77d4f02ba8" (UID: "1969249d-ee44-4685-86de-2a77d4f02ba8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.490432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.490409    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx9hn\" (UniqueName: \"kubernetes.io/projected/42a0b7f7-8c25-4635-8da5-efba3220e87a-kube-api-access-fx9hn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.490432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.490431    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1969249d-ee44-4685-86de-2a77d4f02ba8-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.492107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.492076    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1969249d-ee44-4685-86de-2a77d4f02ba8-kube-api-access-b75g5" pod "1969249d-ee44-4685-86de-2a77d4f02ba8" (UID: "1969249d-ee44-4685-86de-2a77d4f02ba8"). InnerVolumeSpecName "kube-api-access-b75g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:20.581252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.581214    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" path="/var/lib/kubelet/pods/6f5e8754-fd77-46a6-b7a6-d933757dc0a2/volumes"
Oct 30 01:30:20.581828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.581806    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b684507c-883f-403e-9090-16053c2b83fe" path="/var/lib/kubelet/pods/b684507c-883f-403e-9090-16053c2b83fe/volumes"
Oct 30 01:30:20.582254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.582237    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" path="/var/lib/kubelet/pods/e51b9f4c-2235-4095-860f-c0e5f8b7cd8c/volumes"
Oct 30 01:30:20.592064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.592024    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b75g5\" (UniqueName: \"kubernetes.io/projected/1969249d-ee44-4685-86de-2a77d4f02ba8-kube-api-access-b75g5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.599579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.599551    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"]
Oct 30 01:30:20.600705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600673    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:20.600705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600700    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600726    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600735    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600748    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600757    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600770    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600777    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600796    5104 cpu_manager.go:414] "containerMap: removing container" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600803    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600836    5104 cpu_manager.go:414] "containerMap: removing container" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600843    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:20.600868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600868    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600876    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600886    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600893    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600912    5104 cpu_manager.go:414] "containerMap: removing container" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600918    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600934    5104 cpu_manager.go:414] "containerMap: removing container" podUID="121c1ac7-2e84-485a-883f-43db564126c2" containerName="revoke-keys"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600942    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121c1ac7-2e84-485a-883f-43db564126c2" containerName="revoke-keys"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600956    5104 cpu_manager.go:414] "containerMap: removing container" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600963    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600983    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.600991    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601003    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601010    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601021    5104 cpu_manager.go:414] "containerMap: removing container" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601219    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601240    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:20.601433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601248    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601450    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601490    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601503    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601515    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601524    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601533    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601542    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601551    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="121c1ac7-2e84-485a-883f-43db564126c2" containerName="revoke-keys"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601560    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601571    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62ca0a8-b831-4192-a5e7-2e6d4fe6340c" containerName="istio-proxy"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601582    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601593    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601600    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601609    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601618    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601628    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51b9f4c-2235-4095-860f-c0e5f8b7cd8c" containerName="cleanup"
Oct 30 01:30:20.602447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.601641    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:20.606078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.606051    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.623386 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.623299    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:20.623731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.623688    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"]
Oct 30 01:30:20.694535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.694109    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.694535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.694176    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ttt\" (UniqueName: \"kubernetes.io/projected/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-kube-api-access-f5ttt\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.694723 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.694556    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.797764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.797255    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.797764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.797333    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ttt\" (UniqueName: \"kubernetes.io/projected/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-kube-api-access-f5ttt\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.798690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.798113    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.798690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.798528    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.802960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.802896    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.819727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.819409    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ttt\" (UniqueName: \"kubernetes.io/projected/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-kube-api-access-f5ttt\") pod \"maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:20.929243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.929217    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 01:30:20.939553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:20.939378    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 01:30:21.014288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.014244    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aab52d9-16c4-470f-97ac-6adbf20090ee-kube-api-access-8lxxx\" (UniqueName: \"kubernetes.io/projected/7aab52d9-16c4-470f-97ac-6adbf20090ee-kube-api-access-8lxxx\") pod \"7aab52d9-16c4-470f-97ac-6adbf20090ee\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") "
Oct 30 01:30:21.014833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.014405    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-plugins-config-volume\") pod \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") "
Oct 30 01:30:21.014833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.014579    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aab52d9-16c4-470f-97ac-6adbf20090ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aab52d9-16c4-470f-97ac-6adbf20090ee-plugins-config-volume\") pod \"7aab52d9-16c4-470f-97ac-6adbf20090ee\" (UID: \"7aab52d9-16c4-470f-97ac-6adbf20090ee\") "
Oct 30 01:30:21.014833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.014627    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-kube-api-access-24zgq\" (UniqueName: \"kubernetes.io/projected/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-kube-api-access-24zgq\") pod \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\" (UID: \"7cc18e1d-ab8c-411a-b514-2ab57ab74afa\") "
Oct 30 01:30:21.015947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.015796    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-plugins-config-volume" pod "7cc18e1d-ab8c-411a-b514-2ab57ab74afa" (UID: "7cc18e1d-ab8c-411a-b514-2ab57ab74afa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.016512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.016079    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aab52d9-16c4-470f-97ac-6adbf20090ee-plugins-config-volume" pod "7aab52d9-16c4-470f-97ac-6adbf20090ee" (UID: "7aab52d9-16c4-470f-97ac-6adbf20090ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.017870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.017842    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-kube-api-access-24zgq" pod "7cc18e1d-ab8c-411a-b514-2ab57ab74afa" (UID: "7cc18e1d-ab8c-411a-b514-2ab57ab74afa"). InnerVolumeSpecName "kube-api-access-24zgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.019072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.019044    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aab52d9-16c4-470f-97ac-6adbf20090ee-kube-api-access-8lxxx" pod "7aab52d9-16c4-470f-97ac-6adbf20090ee" (UID: "7aab52d9-16c4-470f-97ac-6adbf20090ee"). InnerVolumeSpecName "kube-api-access-8lxxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.044306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.044263    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:21.076352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.076306    5104 generic.go:356] "Generic (PLEG): container finished" podID="7aab52d9-16c4-470f-97ac-6adbf20090ee" containerID="59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c" exitCode=0
Oct 30 01:30:21.076565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.076540    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" event={"ID":"7aab52d9-16c4-470f-97ac-6adbf20090ee","Type":"ContainerDied","Data":"59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c"}
Oct 30 01:30:21.076699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.076664    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4" event={"ID":"7aab52d9-16c4-470f-97ac-6adbf20090ee","Type":"ContainerDied","Data":"bf57876ccc75d8c10c2637b1ccac1742bac9f1914c2a684dfdadeb9a397fe055"}
Oct 30 01:30:21.076768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.076730    5104 scope.go:122] "RemoveContainer" containerID="59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c"
Oct 30 01:30:21.077655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.077627    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"
Oct 30 01:30:21.085440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.085400    5104 generic.go:356] "Generic (PLEG): container finished" podID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerID="a18278711cb09786d64d0bb6379a2451959582533b23ff24aabc05635e2d7415" exitCode=60
Oct 30 01:30:21.085705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.085570    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" event={"ID":"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840","Type":"ContainerDied","Data":"a18278711cb09786d64d0bb6379a2451959582533b23ff24aabc05635e2d7415"}
Oct 30 01:30:21.086173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.086150    5104 scope.go:122] "RemoveContainer" containerID="a18278711cb09786d64d0bb6379a2451959582533b23ff24aabc05635e2d7415"
Oct 30 01:30:21.087487 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:21.086748    5104 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f_odh-ai-gateway-infra(fb0b81d4-dd2c-4140-8ad3-e13a4da9d840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840"
Oct 30 01:30:21.092746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.092708    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn" event={"ID":"42a0b7f7-8c25-4635-8da5-efba3220e87a","Type":"ContainerDied","Data":"169023eef41789a79532bbe7592d1f92c978cb1fb68d545e9ca7732b570a727b"}
Oct 30 01:30:21.094191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.094165    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"
Oct 30 01:30:21.107152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.107117    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"
Oct 30 01:30:21.107781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.107750    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw" event={"ID":"1969249d-ee44-4685-86de-2a77d4f02ba8","Type":"ContainerDied","Data":"2312f27244ea0c64a4a9e11962dc15b0b72c51dbec55cf67edb8cfa2628d1f3e"}
Oct 30 01:30:21.108414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.108391    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"]
Oct 30 01:30:21.116641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.116574    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lxxx\" (UniqueName: \"kubernetes.io/projected/7aab52d9-16c4-470f-97ac-6adbf20090ee-kube-api-access-8lxxx\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.116641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.116609    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.116641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.116624    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aab52d9-16c4-470f-97ac-6adbf20090ee-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.116641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.116636    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24zgq\" (UniqueName: \"kubernetes.io/projected/7cc18e1d-ab8c-411a-b514-2ab57ab74afa-kube-api-access-24zgq\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.117235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.117213    5104 scope.go:122] "RemoveContainer" containerID="59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c"
Oct 30 01:30:21.128310 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:21.123895    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c\": container with ID starting with 59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c not found: ID does not exist" containerID="59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c"
Oct 30 01:30:21.128310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.123947    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c"} err="failed to get container status \"59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c\": rpc error: code = NotFound desc = could not find container \"59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c\": container with ID starting with 59bedb830566898592e57607371d5690631d1ee413516ead99661ffa83ae6e4c not found: ID does not exist"
Oct 30 01:30:21.128310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.123972    5104 scope.go:122] "RemoveContainer" containerID="b0bd4d6996eb7b189552f612717391ee864dc058221159d41848b51d41836aa1"
Oct 30 01:30:21.128310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.128225    5104 generic.go:356] "Generic (PLEG): container finished" podID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" containerID="be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac" exitCode=0
Oct 30 01:30:21.128807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.128327    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" event={"ID":"7cc18e1d-ab8c-411a-b514-2ab57ab74afa","Type":"ContainerDied","Data":"be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac"}
Oct 30 01:30:21.128807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.128379    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj" event={"ID":"7cc18e1d-ab8c-411a-b514-2ab57ab74afa","Type":"ContainerDied","Data":"3af1971a0b885b3dec3ec4b873cb6fb27a0674f38d1e6293d71c9ce27d94f68b"}
Oct 30 01:30:21.128807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.128502    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"
Oct 30 01:30:21.148705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.146553    5104 scope.go:122] "RemoveContainer" containerID="87529c37987085cc3a21bff3f288d89f523a001ed4c64020b0dc6f7c06a6f95e"
Oct 30 01:30:21.201564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.201121    5104 scope.go:122] "RemoveContainer" containerID="7e88e8e9ae6827cfc1ce278690e529e55245f571302059a30a8b753583ef3e73"
Oct 30 01:30:21.228584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.228135    5104 scope.go:122] "RemoveContainer" containerID="be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac"
Oct 30 01:30:21.242047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.241984    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"]
Oct 30 01:30:21.248911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.248863    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d827b7-9659db7ff-d9sj4"]
Oct 30 01:30:21.259980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.259931    5104 scope.go:122] "RemoveContainer" containerID="be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac"
Oct 30 01:30:21.260709 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:21.260628    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac\": container with ID starting with be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac not found: ID does not exist" containerID="be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac"
Oct 30 01:30:21.260709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.260671    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac"} err="failed to get container status \"be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac\": rpc error: code = NotFound desc = could not find container \"be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac\": container with ID starting with be3c9fa423a4e8b0182c563b516c9ec7248b59889d4a5cb77a70bbb4b5a918ac not found: ID does not exist"
Oct 30 01:30:21.438726 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:21.436177    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57db9115_3fdd_4345_b8ae_7eec4a1f12fa.slice/crio-9986686beb86b66f7c26a2e34b07f580602461f98a255bf9281ef9f1397bf078 WatchSource:0}: Error finding container 9986686beb86b66f7c26a2e34b07f580602461f98a255bf9281ef9f1397bf078: Status 404 returned error can't find the container with id 9986686beb86b66f7c26a2e34b07f580602461f98a255bf9281ef9f1397bf078
Oct 30 01:30:21.440847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.440711    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"]
Oct 30 01:30:21.445619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.445493    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-886adb-29888730-8cnjn"]
Oct 30 01:30:21.455649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.455531    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"]
Oct 30 01:30:21.477196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.477130    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"]
Oct 30 01:30:21.481896 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.481781    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d827b7-6d9bd77954-w49tj"]
Oct 30 01:30:21.512564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.512526    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"]
Oct 30 01:30:21.523863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.523822    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c4847c-29888730-7ppzw"]
Oct 30 01:30:21.625911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.625885    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:21.644616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.644581    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:21.680977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.680930    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"]
Oct 30 01:30:21.681999 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.681968    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7aab52d9-16c4-470f-97ac-6adbf20090ee" containerName="payload-pre-processing"
Oct 30 01:30:21.682219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682190    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aab52d9-16c4-470f-97ac-6adbf20090ee" containerName="payload-pre-processing"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682233    5104 cpu_manager.go:414] "containerMap: removing container" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682243    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682256    5104 cpu_manager.go:414] "containerMap: removing container" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682264    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682277    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682285    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682303    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682311    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682336    5104 cpu_manager.go:414] "containerMap: removing container" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:21.682354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682343    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a1808d-54fb-4bdc-865b-e293422c0d29" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682379    5104 cpu_manager.go:414] "containerMap: removing container" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682386    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682398    5104 cpu_manager.go:414] "containerMap: removing container" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682404    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682416    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" containerName="payload-processing"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682423    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" containerName="payload-processing"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682444    5104 cpu_manager.go:414] "containerMap: removing container" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682451    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682482    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682489    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b684507c-883f-403e-9090-16053c2b83fe" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682683    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682699    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682711    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" containerName="payload-processing"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682724    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682735    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5e8754-fd77-46a6-b7a6-d933757dc0a2" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682747    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682760    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aab52d9-16c4-470f-97ac-6adbf20090ee" containerName="payload-pre-processing"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682770    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682781    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:21.682857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.682795    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:21.687513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.687486    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.719943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.710971    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"]
Oct 30 01:30:21.734709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.734560    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-kube-api-access-f44k2\" (UniqueName: \"kubernetes.io/projected/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-kube-api-access-f44k2\") pod \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") "
Oct 30 01:30:21.736433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.735373    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73745591-3ee2-4e22-b43e-453ce48e765a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73745591-3ee2-4e22-b43e-453ce48e765a-maas-api-service-ca\") pod \"73745591-3ee2-4e22-b43e-453ce48e765a\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") "
Oct 30 01:30:21.736433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.735426    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-maas-api-service-ca\") pod \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\" (UID: \"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8\") "
Oct 30 01:30:21.736433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.735471    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73745591-3ee2-4e22-b43e-453ce48e765a-kube-api-access-lr69t\" (UniqueName: \"kubernetes.io/projected/73745591-3ee2-4e22-b43e-453ce48e765a-kube-api-access-lr69t\") pod \"73745591-3ee2-4e22-b43e-453ce48e765a\" (UID: \"73745591-3ee2-4e22-b43e-453ce48e765a\") "
Oct 30 01:30:21.737286 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.737246    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-maas-api-service-ca" pod "079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" (UID: "079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.739880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.738724    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73745591-3ee2-4e22-b43e-453ce48e765a-maas-api-service-ca" pod "73745591-3ee2-4e22-b43e-453ce48e765a" (UID: "73745591-3ee2-4e22-b43e-453ce48e765a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.742505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.742423    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73745591-3ee2-4e22-b43e-453ce48e765a-kube-api-access-lr69t" pod "73745591-3ee2-4e22-b43e-453ce48e765a" (UID: "73745591-3ee2-4e22-b43e-453ce48e765a"). InnerVolumeSpecName "kube-api-access-lr69t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.745733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.745705    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-kube-api-access-f44k2" pod "079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" (UID: "079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8"). InnerVolumeSpecName "kube-api-access-f44k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.837362 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.837319    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlgxb\" (UniqueName: \"kubernetes.io/projected/b4e9679b-3bbb-4c82-a88b-252093291d4d-kube-api-access-wlgxb\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.837591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.837498    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.837591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.837547    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.837730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.837681    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f44k2\" (UniqueName: \"kubernetes.io/projected/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-kube-api-access-f44k2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.837730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.837699    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73745591-3ee2-4e22-b43e-453ce48e765a-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.837730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.837713    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.837730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.837728    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr69t\" (UniqueName: \"kubernetes.io/projected/73745591-3ee2-4e22-b43e-453ce48e765a-kube-api-access-lr69t\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.933138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.933093    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 01:30:21.938662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.938629    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.938852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.938673    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.938852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.938803    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgxb\" (UniqueName: \"kubernetes.io/projected/b4e9679b-3bbb-4c82-a88b-252093291d4d-kube-api-access-wlgxb\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.939724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.939689    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.942423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.942391    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.961709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.961674    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlgxb\" (UniqueName: \"kubernetes.io/projected/b4e9679b-3bbb-4c82-a88b-252093291d4d-kube-api-access-wlgxb\") pod \"maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:21.969930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:21.969904    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 01:30:22.040373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.040327    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbf2e51-6a9c-4902-8438-ae95f266a431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbf2e51-6a9c-4902-8438-ae95f266a431-plugins-config-volume\") pod \"abbf2e51-6a9c-4902-8438-ae95f266a431\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") "
Oct 30 01:30:22.040802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.040490    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbf2e51-6a9c-4902-8438-ae95f266a431-kube-api-access-dkrzn\" (UniqueName: \"kubernetes.io/projected/abbf2e51-6a9c-4902-8438-ae95f266a431-kube-api-access-dkrzn\") pod \"abbf2e51-6a9c-4902-8438-ae95f266a431\" (UID: \"abbf2e51-6a9c-4902-8438-ae95f266a431\") "
Oct 30 01:30:22.040802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.040671    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801aa334-19ec-4ae7-9c34-19bab145c954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801aa334-19ec-4ae7-9c34-19bab145c954-plugins-config-volume\") pod \"801aa334-19ec-4ae7-9c34-19bab145c954\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") "
Oct 30 01:30:22.040802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.040707    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801aa334-19ec-4ae7-9c34-19bab145c954-kube-api-access-pxzpr\" (UniqueName: \"kubernetes.io/projected/801aa334-19ec-4ae7-9c34-19bab145c954-kube-api-access-pxzpr\") pod \"801aa334-19ec-4ae7-9c34-19bab145c954\" (UID: \"801aa334-19ec-4ae7-9c34-19bab145c954\") "
Oct 30 01:30:22.040802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.040792    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbf2e51-6a9c-4902-8438-ae95f266a431-plugins-config-volume" pod "abbf2e51-6a9c-4902-8438-ae95f266a431" (UID: "abbf2e51-6a9c-4902-8438-ae95f266a431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:22.041121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.041090    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801aa334-19ec-4ae7-9c34-19bab145c954-plugins-config-volume" pod "801aa334-19ec-4ae7-9c34-19bab145c954" (UID: "801aa334-19ec-4ae7-9c34-19bab145c954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:22.041275 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.041265    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801aa334-19ec-4ae7-9c34-19bab145c954-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.041341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.041279    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbf2e51-6a9c-4902-8438-ae95f266a431-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.042817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.042790    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbf2e51-6a9c-4902-8438-ae95f266a431-kube-api-access-dkrzn" pod "abbf2e51-6a9c-4902-8438-ae95f266a431" (UID: "abbf2e51-6a9c-4902-8438-ae95f266a431"). InnerVolumeSpecName "kube-api-access-dkrzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:22.042929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.042893    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801aa334-19ec-4ae7-9c34-19bab145c954-kube-api-access-pxzpr" pod "801aa334-19ec-4ae7-9c34-19bab145c954" (UID: "801aa334-19ec-4ae7-9c34-19bab145c954"). InnerVolumeSpecName "kube-api-access-pxzpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:22.101481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.098699    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:22.175510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.159831    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkrzn\" (UniqueName: \"kubernetes.io/projected/abbf2e51-6a9c-4902-8438-ae95f266a431-kube-api-access-dkrzn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.175510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.159879    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxzpr\" (UniqueName: \"kubernetes.io/projected/801aa334-19ec-4ae7-9c34-19bab145c954-kube-api-access-pxzpr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.201175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.201136    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"]
Oct 30 01:30:22.209573 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.209539    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp" event={"ID":"73745591-3ee2-4e22-b43e-453ce48e765a","Type":"ContainerDied","Data":"fd510f1e819b283fcf3bdb97bd7db7ebbc7a1707ae0be5584038f75288f7e668"}
Oct 30 01:30:22.209763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.209604    5104 scope.go:122] "RemoveContainer" containerID="18446c035f18afa9610f4b64b591b7291a0cf1c343343b2bf3b00d9ee3f16264"
Oct 30 01:30:22.210515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.209495    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"
Oct 30 01:30:22.221539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.221504    5104 generic.go:356] "Generic (PLEG): container finished" podID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerID="db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302" exitCode=0
Oct 30 01:30:22.221685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.221616    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" event={"ID":"abbf2e51-6a9c-4902-8438-ae95f266a431","Type":"ContainerDied","Data":"db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302"}
Oct 30 01:30:22.221685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.221669    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4" event={"ID":"abbf2e51-6a9c-4902-8438-ae95f266a431","Type":"ContainerDied","Data":"a7a9c7928af25aab67a5ecb4b548383f1027d5ba090e889f6a6dce8a4546b1ca"}
Oct 30 01:30:22.221788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.221757    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"
Oct 30 01:30:22.231784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.231757    5104 scope.go:122] "RemoveContainer" containerID="db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302"
Oct 30 01:30:22.235015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.234983    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9" event={"ID":"079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8","Type":"ContainerDied","Data":"293e3de6ee7d5658ea24433c10c82f9f7935e33bbb0cd5f85c62b7730440ba39"}
Oct 30 01:30:22.235138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.235018    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"
Oct 30 01:30:22.238261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.238004    5104 generic.go:356] "Generic (PLEG): container finished" podID="801aa334-19ec-4ae7-9c34-19bab145c954" containerID="2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b" exitCode=0
Oct 30 01:30:22.238261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.238053    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"
Oct 30 01:30:22.238261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.238180    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" event={"ID":"801aa334-19ec-4ae7-9c34-19bab145c954","Type":"ContainerDied","Data":"2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b"}
Oct 30 01:30:22.238261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.238217    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6" event={"ID":"801aa334-19ec-4ae7-9c34-19bab145c954","Type":"ContainerDied","Data":"406e409a4ca6736d239d06384da4de126da4545a980323b574d10c4ee530bdfc"}
Oct 30 01:30:22.247729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.245766    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x" event={"ID":"57db9115-3fdd-4345-b8ae-7eec4a1f12fa","Type":"ContainerStarted","Data":"198c8597a5499cc245c595dec6dcdf732fec7532209435164b7c1ab7127b6118"}
Oct 30 01:30:22.247729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.245827    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x" event={"ID":"57db9115-3fdd-4345-b8ae-7eec4a1f12fa","Type":"ContainerStarted","Data":"9986686beb86b66f7c26a2e34b07f580602461f98a255bf9281ef9f1397bf078"}
Oct 30 01:30:22.260691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.255765    5104 scope.go:122] "RemoveContainer" containerID="db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302"
Oct 30 01:30:22.275287 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:22.274836    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302\": container with ID starting with db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302 not found: ID does not exist" containerID="db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302"
Oct 30 01:30:22.275287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.274879    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302"} err="failed to get container status \"db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302\": rpc error: code = NotFound desc = could not find container \"db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302\": container with ID starting with db867ca24ac6de5adb02f1d84a1389e13738bb691eafe6427f199794dd7e5302 not found: ID does not exist"
Oct 30 01:30:22.275287 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.274899    5104 scope.go:122] "RemoveContainer" containerID="c2a26e2bef074560bd026b837363c6d034538e911ae6c6b83b73ef763a299a21"
Oct 30 01:30:22.289268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.288609    5104 scope.go:122] "RemoveContainer" containerID="2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b"
Oct 30 01:30:22.305417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.305318    5104 scope.go:122] "RemoveContainer" containerID="2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b"
Oct 30 01:30:22.305824 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:22.305760    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b\": container with ID starting with 2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b not found: ID does not exist" containerID="2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b"
Oct 30 01:30:22.305824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.305796    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b"} err="failed to get container status \"2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b\": rpc error: code = NotFound desc = could not find container \"2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b\": container with ID starting with 2915b12d07cd621824923f36e884d8a592364d9e2bc3c93445732b008c599b5b not found: ID does not exist"
Oct 30 01:30:22.521236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.521198    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"]
Oct 30 01:30:22.528331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.528300    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4f8a6-29888730-nh2q9"]
Oct 30 01:30:22.535272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.535200    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"]
Oct 30 01:30:22.541665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.541596    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c104b8-68d8ff57f5-sllj6"]
Oct 30 01:30:22.567154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567119    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"]
Oct 30 01:30:22.567927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567904    5104 cpu_manager.go:414] "containerMap: removing container" podUID="801aa334-19ec-4ae7-9c34-19bab145c954" containerName="payload-pre-processing"
Oct 30 01:30:22.567927 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567925    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801aa334-19ec-4ae7-9c34-19bab145c954" containerName="payload-pre-processing"
Oct 30 01:30:22.568092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567957    5104 cpu_manager.go:414] "containerMap: removing container" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:22.568092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567964    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:22.568092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567972    5104 cpu_manager.go:414] "containerMap: removing container" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerName="payload-processing"
Oct 30 01:30:22.568092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567977    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerName="payload-processing"
Oct 30 01:30:22.568092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567988    5104 cpu_manager.go:414] "containerMap: removing container" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:22.568092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.567993    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" containerName="cleanup"
Oct 30 01:30:22.568092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.568095    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" containerName="cleanup"
Oct 30 01:30:22.568385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.568104    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" containerName="payload-processing"
Oct 30 01:30:22.568385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.568117    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="801aa334-19ec-4ae7-9c34-19bab145c954" containerName="payload-pre-processing"
Oct 30 01:30:22.568385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.568290    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:22.571642 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.571623    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.582825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.582796    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8" path="/var/lib/kubelet/pods/079d8eee-d0c4-43c5-89cc-31dc5c6d5ea8/volumes"
Oct 30 01:30:22.583414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.583393    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1969249d-ee44-4685-86de-2a77d4f02ba8" path="/var/lib/kubelet/pods/1969249d-ee44-4685-86de-2a77d4f02ba8/volumes"
Oct 30 01:30:22.583913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.583895    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a0b7f7-8c25-4635-8da5-efba3220e87a" path="/var/lib/kubelet/pods/42a0b7f7-8c25-4635-8da5-efba3220e87a/volumes"
Oct 30 01:30:22.584949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.584929    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aab52d9-16c4-470f-97ac-6adbf20090ee" path="/var/lib/kubelet/pods/7aab52d9-16c4-470f-97ac-6adbf20090ee/volumes"
Oct 30 01:30:22.585521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.585483    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc18e1d-ab8c-411a-b514-2ab57ab74afa" path="/var/lib/kubelet/pods/7cc18e1d-ab8c-411a-b514-2ab57ab74afa/volumes"
Oct 30 01:30:22.586037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.586016    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801aa334-19ec-4ae7-9c34-19bab145c954" path="/var/lib/kubelet/pods/801aa334-19ec-4ae7-9c34-19bab145c954/volumes"
Oct 30 01:30:22.616823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.616789    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-kube-api-access-98kmz\" (UniqueName: \"kubernetes.io/projected/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-kube-api-access-98kmz\") pod \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") "
Oct 30 01:30:22.617018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.616911    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-maas-api-service-ca\") pod \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\" (UID: \"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840\") "
Oct 30 01:30:22.617210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.617193    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.617360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.617336    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.617959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.617894    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-maas-api-service-ca" pod "fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" (UID: "fb0b81d4-dd2c-4140-8ad3-e13a4da9d840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:22.618116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.618015    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkwp\" (UniqueName: \"kubernetes.io/projected/430994cb-e380-4b7b-9f56-4bdec2053601-kube-api-access-dzkwp\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.618371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.618345    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-maas-api-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.620722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.620694    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-kube-api-access-98kmz" pod "fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" (UID: "fb0b81d4-dd2c-4140-8ad3-e13a4da9d840"). InnerVolumeSpecName "kube-api-access-98kmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:22.622724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.622701    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"]
Oct 30 01:30:22.624047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.623806    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.624047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.623828    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.624047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.623863    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.624047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.623872    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.624047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.623928    5104 cpu_manager.go:414] "containerMap: removing container" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.624047 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.623937    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.624948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.624098    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.624948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.624116    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:22.627696 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:22.627658    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e9679b_3bbb_4c82_a88b_252093291d4d.slice/crio-fe475d4415e1eecab5bf20bfa98a782c4bd14b8d749077c8154e017f931ee948 WatchSource:0}: Error finding container fe475d4415e1eecab5bf20bfa98a782c4bd14b8d749077c8154e017f931ee948: Status 404 returned error can't find the container with id fe475d4415e1eecab5bf20bfa98a782c4bd14b8d749077c8154e017f931ee948
Oct 30 01:30:22.636093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.636049    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"]
Oct 30 01:30:22.636255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.636115    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"]
Oct 30 01:30:22.636255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.636137    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"]
Oct 30 01:30:22.637423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.636246    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.660957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.660882    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"]
Oct 30 01:30:22.674721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.674675    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c104b8-6c778f644d-bhmp4"]
Oct 30 01:30:22.696551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.696411    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x" podStartSLOduration=2.6963722839999997 podStartE2EDuration="2.696372284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:22.692415695 +0000 UTC m=+2864.240106192" watchObservedRunningTime="2026-10-30 01:30:22.696372284 +0000 UTC m=+2864.244062781"
Oct 30 01:30:22.719385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.719352    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.719606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.719402    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.719606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.719558    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqs52\" (UniqueName: \"kubernetes.io/projected/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-kube-api-access-dqs52\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.719758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.719696    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.719758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.719742    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkwp\" (UniqueName: \"kubernetes.io/projected/430994cb-e380-4b7b-9f56-4bdec2053601-kube-api-access-dzkwp\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.719902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.719887    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.720051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.720036    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98kmz\" (UniqueName: \"kubernetes.io/projected/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840-kube-api-access-98kmz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.720319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.720299    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.721982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.721965    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.739570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.739539    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"]
Oct 30 01:30:22.747379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.747348    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-679bd2-29888730-6bjlp"]
Oct 30 01:30:22.752690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.752654    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkwp\" (UniqueName: \"kubernetes.io/projected/430994cb-e380-4b7b-9f56-4bdec2053601-kube-api-access-dzkwp\") pod \"maas-api-revoke-sub-386e3c17-tdzgq\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:22.821137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.821028    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.821335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.821159    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.821335 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.821221    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqs52\" (UniqueName: \"kubernetes.io/projected/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-kube-api-access-dqs52\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.821961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.821931    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.823827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.823797    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.851373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.851340    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqs52\" (UniqueName: \"kubernetes.io/projected/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-kube-api-access-dqs52\") pod \"maas-api-revoke-sub-139ef9af-mdm9l\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:22.881691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.881640    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"]
Oct 30 01:30:22.886509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.886473    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fd3281-s2fxv"]
Oct 30 01:30:22.954768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:22.954723    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:23.056946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.056897    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:23.258841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.258797    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t" event={"ID":"b4e9679b-3bbb-4c82-a88b-252093291d4d","Type":"ContainerStarted","Data":"7f5c22bbe90de3de376773bd3c76886770b2b1651b4dc4426cd68c28ef5b9817"}
Oct 30 01:30:23.259040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.258852    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t" event={"ID":"b4e9679b-3bbb-4c82-a88b-252093291d4d","Type":"ContainerStarted","Data":"fe475d4415e1eecab5bf20bfa98a782c4bd14b8d749077c8154e017f931ee948"}
Oct 30 01:30:23.260638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.260604    5104 generic.go:356] "Generic (PLEG): container finished" podID="57db9115-3fdd-4345-b8ae-7eec4a1f12fa" containerID="198c8597a5499cc245c595dec6dcdf732fec7532209435164b7c1ab7127b6118" exitCode=0
Oct 30 01:30:23.260799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.260690    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x" event={"ID":"57db9115-3fdd-4345-b8ae-7eec4a1f12fa","Type":"ContainerDied","Data":"198c8597a5499cc245c595dec6dcdf732fec7532209435164b7c1ab7127b6118"}
Oct 30 01:30:23.263503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.263477    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f" event={"ID":"fb0b81d4-dd2c-4140-8ad3-e13a4da9d840","Type":"ContainerDied","Data":"4e8b4efbfd25ce3f5b0df2894c0f7f69ec573f4f1d822d7e3862f82322c4b1ab"}
Oct 30 01:30:23.263503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.263498    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"
Oct 30 01:30:23.263668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.263525    5104 scope.go:122] "RemoveContainer" containerID="a18278711cb09786d64d0bb6379a2451959582533b23ff24aabc05635e2d7415"
Oct 30 01:30:23.300530 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.300496    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"]
Oct 30 01:30:23.302521 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:23.302438    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430994cb_e380_4b7b_9f56_4bdec2053601.slice/crio-a90b852d38465b16b35532150bc2570cdbb6e8c25f4e4c223489965797a399c5 WatchSource:0}: Error finding container a90b852d38465b16b35532150bc2570cdbb6e8c25f4e4c223489965797a399c5: Status 404 returned error can't find the container with id a90b852d38465b16b35532150bc2570cdbb6e8c25f4e4c223489965797a399c5
Oct 30 01:30:23.310236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.310183    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t" podStartSLOduration=2.310164805 podStartE2EDuration="2.310164805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:23.305779957 +0000 UTC m=+2864.853470454" watchObservedRunningTime="2026-10-30 01:30:23.310164805 +0000 UTC m=+2864.857855302"
Oct 30 01:30:23.392528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.392500    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"]
Oct 30 01:30:23.406397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.406340    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"]
Oct 30 01:30:23.410762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:23.410734    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-29bb56-29888730-bvx2f"]
Oct 30 01:30:23.475193 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:23.475148    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a06c24_ebaf_43d1_bc51_3046fb9d6c04.slice/crio-4ad3467b8ae63675d9f99b95fdd6ce9bf8b83e4c4205f3a85abb3249c7e7a9ff WatchSource:0}: Error finding container 4ad3467b8ae63675d9f99b95fdd6ce9bf8b83e4c4205f3a85abb3249c7e7a9ff: Status 404 returned error can't find the container with id 4ad3467b8ae63675d9f99b95fdd6ce9bf8b83e4c4205f3a85abb3249c7e7a9ff
Oct 30 01:30:24.279341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.279304    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l" event={"ID":"36a06c24-ebaf-43d1-bc51-3046fb9d6c04","Type":"ContainerStarted","Data":"3ba5200f9e3380de1280268f8050a04648224dc4cf8268df2d479bfcf408ca13"}
Oct 30 01:30:24.279876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.279366    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l" event={"ID":"36a06c24-ebaf-43d1-bc51-3046fb9d6c04","Type":"ContainerStarted","Data":"4ad3467b8ae63675d9f99b95fdd6ce9bf8b83e4c4205f3a85abb3249c7e7a9ff"}
Oct 30 01:30:24.281574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.281544    5104 generic.go:356] "Generic (PLEG): container finished" podID="430994cb-e380-4b7b-9f56-4bdec2053601" containerID="101636d0832e3c540306037b924daadb43d39532b29b9d3d12e368dbfc8546bf" exitCode=0
Oct 30 01:30:24.281722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.281682    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq" event={"ID":"430994cb-e380-4b7b-9f56-4bdec2053601","Type":"ContainerDied","Data":"101636d0832e3c540306037b924daadb43d39532b29b9d3d12e368dbfc8546bf"}
Oct 30 01:30:24.281767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.281729    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq" event={"ID":"430994cb-e380-4b7b-9f56-4bdec2053601","Type":"ContainerStarted","Data":"a90b852d38465b16b35532150bc2570cdbb6e8c25f4e4c223489965797a399c5"}
Oct 30 01:30:24.284421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.284395    5104 generic.go:356] "Generic (PLEG): container finished" podID="b4e9679b-3bbb-4c82-a88b-252093291d4d" containerID="7f5c22bbe90de3de376773bd3c76886770b2b1651b4dc4426cd68c28ef5b9817" exitCode=0
Oct 30 01:30:24.284616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.284600    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t" event={"ID":"b4e9679b-3bbb-4c82-a88b-252093291d4d","Type":"ContainerDied","Data":"7f5c22bbe90de3de376773bd3c76886770b2b1651b4dc4426cd68c28ef5b9817"}
Oct 30 01:30:24.320890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.320819    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l" podStartSLOduration=2.320797583 podStartE2EDuration="2.320797583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:24.316365988 +0000 UTC m=+2865.864056486" watchObservedRunningTime="2026-10-30 01:30:24.320797583 +0000 UTC m=+2865.868488081"
Oct 30 01:30:24.474277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.474218    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:24.546160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.546118    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-kube-api-access-f5ttt\" (UniqueName: \"kubernetes.io/projected/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-kube-api-access-f5ttt\") pod \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") "
Oct 30 01:30:24.546363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.546285    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-xks-ca\") pod \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") "
Oct 30 01:30:24.546432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.546397    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-openshift-ca\") pod \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\" (UID: \"57db9115-3fdd-4345-b8ae-7eec4a1f12fa\") "
Oct 30 01:30:24.547341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.547312    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-openshift-ca" pod "57db9115-3fdd-4345-b8ae-7eec4a1f12fa" (UID: "57db9115-3fdd-4345-b8ae-7eec4a1f12fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:24.549931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.549902    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-kube-api-access-f5ttt" pod "57db9115-3fdd-4345-b8ae-7eec4a1f12fa" (UID: "57db9115-3fdd-4345-b8ae-7eec4a1f12fa"). InnerVolumeSpecName "kube-api-access-f5ttt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:24.550167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.550145    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-xks-ca" pod "57db9115-3fdd-4345-b8ae-7eec4a1f12fa" (UID: "57db9115-3fdd-4345-b8ae-7eec4a1f12fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:24.582726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.582683    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73745591-3ee2-4e22-b43e-453ce48e765a" path="/var/lib/kubelet/pods/73745591-3ee2-4e22-b43e-453ce48e765a/volumes"
Oct 30 01:30:24.583271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.583250    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29f43b8-6662-418c-94ea-c0be9254013f" path="/var/lib/kubelet/pods/a29f43b8-6662-418c-94ea-c0be9254013f/volumes"
Oct 30 01:30:24.583718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.583691    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbf2e51-6a9c-4902-8438-ae95f266a431" path="/var/lib/kubelet/pods/abbf2e51-6a9c-4902-8438-ae95f266a431/volumes"
Oct 30 01:30:24.584435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.584418    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" path="/var/lib/kubelet/pods/fb0b81d4-dd2c-4140-8ad3-e13a4da9d840/volumes"
Oct 30 01:30:24.648681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.648636    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.648681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.648666    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5ttt\" (UniqueName: \"kubernetes.io/projected/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-kube-api-access-f5ttt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.648681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:24.648679    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57db9115-3fdd-4345-b8ae-7eec4a1f12fa-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:25.297405 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.297364    5104 generic.go:356] "Generic (PLEG): container finished" podID="36a06c24-ebaf-43d1-bc51-3046fb9d6c04" containerID="3ba5200f9e3380de1280268f8050a04648224dc4cf8268df2d479bfcf408ca13" exitCode=0
Oct 30 01:30:25.297869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.297443    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l" event={"ID":"36a06c24-ebaf-43d1-bc51-3046fb9d6c04","Type":"ContainerDied","Data":"3ba5200f9e3380de1280268f8050a04648224dc4cf8268df2d479bfcf408ca13"}
Oct 30 01:30:25.299220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.299200    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"
Oct 30 01:30:25.299348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.299228    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x" event={"ID":"57db9115-3fdd-4345-b8ae-7eec4a1f12fa","Type":"ContainerDied","Data":"9986686beb86b66f7c26a2e34b07f580602461f98a255bf9281ef9f1397bf078"}
Oct 30 01:30:25.299348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.299278    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9986686beb86b66f7c26a2e34b07f580602461f98a255bf9281ef9f1397bf078"
Oct 30 01:30:25.509330 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.509297    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:25.556685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.556624    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:25.564096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.564066    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e9679b-3bbb-4c82-a88b-252093291d4d-kube-api-access-wlgxb\" (UniqueName: \"kubernetes.io/projected/b4e9679b-3bbb-4c82-a88b-252093291d4d-kube-api-access-wlgxb\") pod \"b4e9679b-3bbb-4c82-a88b-252093291d4d\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") "
Oct 30 01:30:25.564258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.564141    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-openshift-ca\") pod \"b4e9679b-3bbb-4c82-a88b-252093291d4d\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") "
Oct 30 01:30:25.564404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.564386    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-xks-ca\") pod \"b4e9679b-3bbb-4c82-a88b-252093291d4d\" (UID: \"b4e9679b-3bbb-4c82-a88b-252093291d4d\") "
Oct 30 01:30:25.564802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.564584    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-openshift-ca" pod "b4e9679b-3bbb-4c82-a88b-252093291d4d" (UID: "b4e9679b-3bbb-4c82-a88b-252093291d4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:25.565044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.565018    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:25.567022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.566999    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e9679b-3bbb-4c82-a88b-252093291d4d-kube-api-access-wlgxb" pod "b4e9679b-3bbb-4c82-a88b-252093291d4d" (UID: "b4e9679b-3bbb-4c82-a88b-252093291d4d"). InnerVolumeSpecName "kube-api-access-wlgxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:25.567114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.567028    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-xks-ca" pod "b4e9679b-3bbb-4c82-a88b-252093291d4d" (UID: "b4e9679b-3bbb-4c82-a88b-252093291d4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:25.665541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.665503    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-openshift-ca\") pod \"430994cb-e380-4b7b-9f56-4bdec2053601\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") "
Oct 30 01:30:25.665743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.665548    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430994cb-e380-4b7b-9f56-4bdec2053601-kube-api-access-dzkwp\" (UniqueName: \"kubernetes.io/projected/430994cb-e380-4b7b-9f56-4bdec2053601-kube-api-access-dzkwp\") pod \"430994cb-e380-4b7b-9f56-4bdec2053601\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") "
Oct 30 01:30:25.665743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.665664    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-xks-ca\") pod \"430994cb-e380-4b7b-9f56-4bdec2053601\" (UID: \"430994cb-e380-4b7b-9f56-4bdec2053601\") "
Oct 30 01:30:25.665990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.665959    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-openshift-ca" pod "430994cb-e380-4b7b-9f56-4bdec2053601" (UID: "430994cb-e380-4b7b-9f56-4bdec2053601"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:25.667163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.667134    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:25.667378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.667359    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e9679b-3bbb-4c82-a88b-252093291d4d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:25.667516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.667499    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlgxb\" (UniqueName: \"kubernetes.io/projected/b4e9679b-3bbb-4c82-a88b-252093291d4d-kube-api-access-wlgxb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:25.668407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.668340    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430994cb-e380-4b7b-9f56-4bdec2053601-kube-api-access-dzkwp" pod "430994cb-e380-4b7b-9f56-4bdec2053601" (UID: "430994cb-e380-4b7b-9f56-4bdec2053601"). InnerVolumeSpecName "kube-api-access-dzkwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:25.673380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.673355    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-xks-ca" pod "430994cb-e380-4b7b-9f56-4bdec2053601" (UID: "430994cb-e380-4b7b-9f56-4bdec2053601"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:25.768522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.768452    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkwp\" (UniqueName: \"kubernetes.io/projected/430994cb-e380-4b7b-9f56-4bdec2053601-kube-api-access-dzkwp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:25.768522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:25.768518    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430994cb-e380-4b7b-9f56-4bdec2053601-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:26.310492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.310447    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"
Oct 30 01:30:26.310903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.310487    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq" event={"ID":"430994cb-e380-4b7b-9f56-4bdec2053601","Type":"ContainerDied","Data":"a90b852d38465b16b35532150bc2570cdbb6e8c25f4e4c223489965797a399c5"}
Oct 30 01:30:26.310903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.310553    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a90b852d38465b16b35532150bc2570cdbb6e8c25f4e4c223489965797a399c5"
Oct 30 01:30:26.312520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.312495    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"
Oct 30 01:30:26.312694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.312495    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t" event={"ID":"b4e9679b-3bbb-4c82-a88b-252093291d4d","Type":"ContainerDied","Data":"fe475d4415e1eecab5bf20bfa98a782c4bd14b8d749077c8154e017f931ee948"}
Oct 30 01:30:26.312694 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.312616    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe475d4415e1eecab5bf20bfa98a782c4bd14b8d749077c8154e017f931ee948"
Oct 30 01:30:26.509024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.508986    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:26.583102 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.582994    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-kube-api-access-dqs52\" (UniqueName: \"kubernetes.io/projected/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-kube-api-access-dqs52\") pod \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") "
Oct 30 01:30:26.583299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.583103    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-openshift-ca\") pod \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") "
Oct 30 01:30:26.583374 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.583342    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-xks-ca\") pod \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\" (UID: \"36a06c24-ebaf-43d1-bc51-3046fb9d6c04\") "
Oct 30 01:30:26.584894 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.584860    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-openshift-ca" pod "36a06c24-ebaf-43d1-bc51-3046fb9d6c04" (UID: "36a06c24-ebaf-43d1-bc51-3046fb9d6c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:26.586931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.586889    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-xks-ca" pod "36a06c24-ebaf-43d1-bc51-3046fb9d6c04" (UID: "36a06c24-ebaf-43d1-bc51-3046fb9d6c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:26.595828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.595773    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-kube-api-access-dqs52" pod "36a06c24-ebaf-43d1-bc51-3046fb9d6c04" (UID: "36a06c24-ebaf-43d1-bc51-3046fb9d6c04"). InnerVolumeSpecName "kube-api-access-dqs52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:26.685388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.685348    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:26.685388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.685390    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqs52\" (UniqueName: \"kubernetes.io/projected/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-kube-api-access-dqs52\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:26.685626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:26.685407    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a06c24-ebaf-43d1-bc51-3046fb9d6c04-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:27.322638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:27.322608    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"
Oct 30 01:30:27.323022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:27.322648    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l" event={"ID":"36a06c24-ebaf-43d1-bc51-3046fb9d6c04","Type":"ContainerDied","Data":"4ad3467b8ae63675d9f99b95fdd6ce9bf8b83e4c4205f3a85abb3249c7e7a9ff"}
Oct 30 01:30:27.323022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:27.322695    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ad3467b8ae63675d9f99b95fdd6ce9bf8b83e4c4205f3a85abb3249c7e7a9ff"
Oct 30 01:30:28.480578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.480533    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main" containerID="cri-o://364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577" gracePeriod=45
Oct 30 01:30:28.779382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.779346    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:30:28.920114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920079    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-home\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-home\") pod \"3c35be6b-f3d3-4394-9e50-ff646a452e86\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") "
Oct 30 01:30:28.920278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920158    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c35be6b-f3d3-4394-9e50-ff646a452e86-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c35be6b-f3d3-4394-9e50-ff646a452e86-tls-certs\") pod \"3c35be6b-f3d3-4394-9e50-ff646a452e86\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") "
Oct 30 01:30:28.920324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920282    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-dshm\") pod \"3c35be6b-f3d3-4394-9e50-ff646a452e86\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") "
Oct 30 01:30:28.920406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920360    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c35be6b-f3d3-4394-9e50-ff646a452e86-kube-api-access-lzpvn\" (UniqueName: \"kubernetes.io/projected/3c35be6b-f3d3-4394-9e50-ff646a452e86-kube-api-access-lzpvn\") pod \"3c35be6b-f3d3-4394-9e50-ff646a452e86\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") "
Oct 30 01:30:28.920406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920376    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-home" pod "3c35be6b-f3d3-4394-9e50-ff646a452e86" (UID: "3c35be6b-f3d3-4394-9e50-ff646a452e86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:28.920542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920406    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-model-cache\") pod \"3c35be6b-f3d3-4394-9e50-ff646a452e86\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") "
Oct 30 01:30:28.920630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920591    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-tmp-dir\") pod \"3c35be6b-f3d3-4394-9e50-ff646a452e86\" (UID: \"3c35be6b-f3d3-4394-9e50-ff646a452e86\") "
Oct 30 01:30:28.920765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920738    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-model-cache" pod "3c35be6b-f3d3-4394-9e50-ff646a452e86" (UID: "3c35be6b-f3d3-4394-9e50-ff646a452e86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:28.920918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.920892    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-tmp-dir" pod "3c35be6b-f3d3-4394-9e50-ff646a452e86" (UID: "3c35be6b-f3d3-4394-9e50-ff646a452e86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:28.921299 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.921276    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:28.921392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.921301    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:28.921392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.921315    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:28.922707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.922680    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-dshm" pod "3c35be6b-f3d3-4394-9e50-ff646a452e86" (UID: "3c35be6b-f3d3-4394-9e50-ff646a452e86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:28.922865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.922842    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35be6b-f3d3-4394-9e50-ff646a452e86-kube-api-access-lzpvn" pod "3c35be6b-f3d3-4394-9e50-ff646a452e86" (UID: "3c35be6b-f3d3-4394-9e50-ff646a452e86"). InnerVolumeSpecName "kube-api-access-lzpvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:28.923307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:28.923282    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c35be6b-f3d3-4394-9e50-ff646a452e86-tls-certs" pod "3c35be6b-f3d3-4394-9e50-ff646a452e86" (UID: "3c35be6b-f3d3-4394-9e50-ff646a452e86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:29.022764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.022674    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzpvn\" (UniqueName: \"kubernetes.io/projected/3c35be6b-f3d3-4394-9e50-ff646a452e86-kube-api-access-lzpvn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:29.022764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.022710    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c35be6b-f3d3-4394-9e50-ff646a452e86-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:29.022764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.022722    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c35be6b-f3d3-4394-9e50-ff646a452e86-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:29.444972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.444936    5104 generic.go:356] "Generic (PLEG): container finished" podID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerID="364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577" exitCode=0
Oct 30 01:30:29.445340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.445188    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" event={"ID":"3c35be6b-f3d3-4394-9e50-ff646a452e86","Type":"ContainerDied","Data":"364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577"}
Oct 30 01:30:29.445340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.445228    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7" event={"ID":"3c35be6b-f3d3-4394-9e50-ff646a452e86","Type":"ContainerDied","Data":"6a3f47d175ad5f141f06bf1bf3e8e0b59d6894d5d816c966f623dd978066ed34"}
Oct 30 01:30:29.445340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.445246    5104 scope.go:122] "RemoveContainer" containerID="364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577"
Oct 30 01:30:29.445548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.445393    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"
Oct 30 01:30:29.460613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.460575    5104 scope.go:122] "RemoveContainer" containerID="364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577"
Oct 30 01:30:29.461348 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:29.461310    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577\": container with ID starting with 364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577 not found: ID does not exist" containerID="364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577"
Oct 30 01:30:29.461536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.461360    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577"} err="failed to get container status \"364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577\": rpc error: code = NotFound desc = could not find container \"364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577\": container with ID starting with 364e1b795ff591afa0260170db4db47d562f2297fe5b2caaba1e752998265577 not found: ID does not exist"
Oct 30 01:30:29.510676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.510634    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"]
Oct 30 01:30:29.519169 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.519136    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/simulator-w5-dca38d-kserve-5b878fd8cf-tprg7"]
Oct 30 01:30:29.914650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.914534    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"]
Oct 30 01:30:29.919867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:29.919229    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d827b7-nhfvx"]
Oct 30 01:30:30.578531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:30.578494    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" path="/var/lib/kubelet/pods/3c35be6b-f3d3-4394-9e50-ff646a452e86/volumes"
Oct 30 01:30:30.578980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:30.578965    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33" path="/var/lib/kubelet/pods/67c0f7a1-a2a0-4b6c-a0ba-7cc12f88ca33/volumes"
Oct 30 01:30:32.765897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:32.765854    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"]
Oct 30 01:30:32.771119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:32.771071    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c104b8-sp2fq"]
Oct 30 01:30:34.579346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:34.579308    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f451ba8b-697a-4bea-88bf-b63e4e1c9690" path="/var/lib/kubelet/pods/f451ba8b-697a-4bea-88bf-b63e4e1c9690/volumes"
Oct 30 01:30:34.685781 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:34.685743    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:30:34.941015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:34.940972    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"]
Oct 30 01:30:34.941564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:34.941522    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy" containerID="cri-o://1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651" gracePeriod=30
Oct 30 01:30:38.494578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.494537    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"]
Oct 30 01:30:38.494985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.494757    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s" podUID="d62267d3-7fa0-436e-a6fb-bf197972e269" containerName="maas-api" containerID="cri-o://039e32a6a5d40f1710f832cd421440b987a1792c49af3424b341cce0995dc5ce" gracePeriod=30
Oct 30 01:30:38.638092 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.638053    5104 generic.go:356] "Generic (PLEG): container finished" podID="d62267d3-7fa0-436e-a6fb-bf197972e269" containerID="039e32a6a5d40f1710f832cd421440b987a1792c49af3424b341cce0995dc5ce" exitCode=0
Oct 30 01:30:38.638282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.638124    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s" event={"ID":"d62267d3-7fa0-436e-a6fb-bf197972e269","Type":"ContainerDied","Data":"039e32a6a5d40f1710f832cd421440b987a1792c49af3424b341cce0995dc5ce"}
Oct 30 01:30:38.711316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.711000    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"]
Oct 30 01:30:38.711316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.711272    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9" podUID="6768328a-78ee-4d01-9e6c-df92c1c6a8d4" containerName="payload-processing" containerID="cri-o://1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506" gracePeriod=30
Oct 30 01:30:38.739433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.739321    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"]
Oct 30 01:30:38.739636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.739605    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p" podUID="de81acc9-6f4e-4d26-81ff-50652d037811" containerName="payload-pre-processing" containerID="cri-o://110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f" gracePeriod=30
Oct 30 01:30:38.883539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.883420    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:30:38.970847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.970809    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62267d3-7fa0-436e-a6fb-bf197972e269-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62267d3-7fa0-436e-a6fb-bf197972e269-openshift-service-ca\") pod \"d62267d3-7fa0-436e-a6fb-bf197972e269\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") "
Oct 30 01:30:38.971037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.970861    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-maas-api-tls\") pod \"d62267d3-7fa0-436e-a6fb-bf197972e269\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") "
Oct 30 01:30:38.971037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.970929    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-metrics-certs\") pod \"d62267d3-7fa0-436e-a6fb-bf197972e269\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") "
Oct 30 01:30:38.971037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.970963    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62267d3-7fa0-436e-a6fb-bf197972e269-kube-api-access-5lkzd\" (UniqueName: \"kubernetes.io/projected/d62267d3-7fa0-436e-a6fb-bf197972e269-kube-api-access-5lkzd\") pod \"d62267d3-7fa0-436e-a6fb-bf197972e269\" (UID: \"d62267d3-7fa0-436e-a6fb-bf197972e269\") "
Oct 30 01:30:38.971876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.971821    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62267d3-7fa0-436e-a6fb-bf197972e269-openshift-service-ca" pod "d62267d3-7fa0-436e-a6fb-bf197972e269" (UID: "d62267d3-7fa0-436e-a6fb-bf197972e269"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:38.977517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.977425    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62267d3-7fa0-436e-a6fb-bf197972e269-kube-api-access-5lkzd" pod "d62267d3-7fa0-436e-a6fb-bf197972e269" (UID: "d62267d3-7fa0-436e-a6fb-bf197972e269"). InnerVolumeSpecName "kube-api-access-5lkzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:38.978852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.978810    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-maas-api-tls" pod "d62267d3-7fa0-436e-a6fb-bf197972e269" (UID: "d62267d3-7fa0-436e-a6fb-bf197972e269"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:38.981774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:38.981732    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-metrics-certs" pod "d62267d3-7fa0-436e-a6fb-bf197972e269" (UID: "d62267d3-7fa0-436e-a6fb-bf197972e269"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:39.034410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.034371    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:30:39.055099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.054473    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:30:39.098627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.097910    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-plugins-config-volume\") pod \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") "
Oct 30 01:30:39.098627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.097971    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-kube-api-access-mlgvf\" (UniqueName: \"kubernetes.io/projected/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-kube-api-access-mlgvf\") pod \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\" (UID: \"6768328a-78ee-4d01-9e6c-df92c1c6a8d4\") "
Oct 30 01:30:39.098627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.098281    5104 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62267d3-7fa0-436e-a6fb-bf197972e269-openshift-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.098627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.098294    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-maas-api-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.098627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.098305    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62267d3-7fa0-436e-a6fb-bf197972e269-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.098627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.098313    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkzd\" (UniqueName: \"kubernetes.io/projected/d62267d3-7fa0-436e-a6fb-bf197972e269-kube-api-access-5lkzd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.103101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.103035    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-kube-api-access-mlgvf" pod "6768328a-78ee-4d01-9e6c-df92c1c6a8d4" (UID: "6768328a-78ee-4d01-9e6c-df92c1c6a8d4"). InnerVolumeSpecName "kube-api-access-mlgvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:39.104421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.103996    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-plugins-config-volume" pod "6768328a-78ee-4d01-9e6c-df92c1c6a8d4" (UID: "6768328a-78ee-4d01-9e6c-df92c1c6a8d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:39.169849 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:39.169804    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:30:39.199614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.199580    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de81acc9-6f4e-4d26-81ff-50652d037811-kube-api-access-n5266\" (UniqueName: \"kubernetes.io/projected/de81acc9-6f4e-4d26-81ff-50652d037811-kube-api-access-n5266\") pod \"de81acc9-6f4e-4d26-81ff-50652d037811\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") "
Oct 30 01:30:39.199823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.199788    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de81acc9-6f4e-4d26-81ff-50652d037811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de81acc9-6f4e-4d26-81ff-50652d037811-plugins-config-volume\") pod \"de81acc9-6f4e-4d26-81ff-50652d037811\" (UID: \"de81acc9-6f4e-4d26-81ff-50652d037811\") "
Oct 30 01:30:39.200137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.200117    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.200137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.200135    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlgvf\" (UniqueName: \"kubernetes.io/projected/6768328a-78ee-4d01-9e6c-df92c1c6a8d4-kube-api-access-mlgvf\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.200327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.200136    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de81acc9-6f4e-4d26-81ff-50652d037811-plugins-config-volume" pod "de81acc9-6f4e-4d26-81ff-50652d037811" (UID: "de81acc9-6f4e-4d26-81ff-50652d037811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:39.201941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.201916    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de81acc9-6f4e-4d26-81ff-50652d037811-kube-api-access-n5266" pod "de81acc9-6f4e-4d26-81ff-50652d037811" (UID: "de81acc9-6f4e-4d26-81ff-50652d037811"). InnerVolumeSpecName "kube-api-access-n5266". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:39.237741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.237698    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"]
Oct 30 01:30:39.238011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.237981    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b" podUID="1863d77a-baf2-4c78-bfe0-9b9b15bf319e" containerName="payload-processing" containerID="cri-o://470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78" gracePeriod=30
Oct 30 01:30:39.254529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.254492    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"]
Oct 30 01:30:39.254817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.254780    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2" podUID="93218cf5-525b-4b93-8ef4-63c8ca95408d" containerName="payload-pre-processing" containerID="cri-o://07e5c3a190c79a42a8203f601424914d8344f038d523b722809b6fed1be4c941" gracePeriod=30
Oct 30 01:30:39.301900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.301857    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de81acc9-6f4e-4d26-81ff-50652d037811-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.302089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.301912    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5266\" (UniqueName: \"kubernetes.io/projected/de81acc9-6f4e-4d26-81ff-50652d037811-kube-api-access-n5266\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.396969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.396902    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"]
Oct 30 01:30:39.402041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.402005    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eaf753-ztx9t"]
Oct 30 01:30:39.678560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.678523    5104 generic.go:356] "Generic (PLEG): container finished" podID="93218cf5-525b-4b93-8ef4-63c8ca95408d" containerID="07e5c3a190c79a42a8203f601424914d8344f038d523b722809b6fed1be4c941" exitCode=0
Oct 30 01:30:39.679087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.678596    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2" event={"ID":"93218cf5-525b-4b93-8ef4-63c8ca95408d","Type":"ContainerDied","Data":"07e5c3a190c79a42a8203f601424914d8344f038d523b722809b6fed1be4c941"}
Oct 30 01:30:39.680715 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.680383    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:30:39.680954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.680932    5104 generic.go:356] "Generic (PLEG): container finished" podID="6768328a-78ee-4d01-9e6c-df92c1c6a8d4" containerID="1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506" exitCode=0
Oct 30 01:30:39.681316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.681096    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9" event={"ID":"6768328a-78ee-4d01-9e6c-df92c1c6a8d4","Type":"ContainerDied","Data":"1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506"}
Oct 30 01:30:39.681316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.681156    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9" event={"ID":"6768328a-78ee-4d01-9e6c-df92c1c6a8d4","Type":"ContainerDied","Data":"9aebb30b16e1d785bd133c5bfbe4d3a29bb338bf7b35901cd61245502e6e2aa4"}
Oct 30 01:30:39.681316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.681188    5104 scope.go:122] "RemoveContainer" containerID="1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506"
Oct 30 01:30:39.681951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.681931    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"
Oct 30 01:30:39.684562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.684352    5104 generic.go:356] "Generic (PLEG): container finished" podID="de81acc9-6f4e-4d26-81ff-50652d037811" containerID="110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f" exitCode=0
Oct 30 01:30:39.685431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.684444    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p" event={"ID":"de81acc9-6f4e-4d26-81ff-50652d037811","Type":"ContainerDied","Data":"110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f"}
Oct 30 01:30:39.685431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.684743    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p" event={"ID":"de81acc9-6f4e-4d26-81ff-50652d037811","Type":"ContainerDied","Data":"3d67c00b2d5759318ecfd9c8ff2dd4e3d334e1d45f1784d603edd9a7f00cccf5"}
Oct 30 01:30:39.685431 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.684843    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"
Oct 30 01:30:39.696278 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.696242    5104 generic.go:356] "Generic (PLEG): container finished" podID="1863d77a-baf2-4c78-bfe0-9b9b15bf319e" containerID="470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78" exitCode=0
Oct 30 01:30:39.699778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.696373    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"
Oct 30 01:30:39.699778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.696580    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b" event={"ID":"1863d77a-baf2-4c78-bfe0-9b9b15bf319e","Type":"ContainerDied","Data":"470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78"}
Oct 30 01:30:39.699778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.696634    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b" event={"ID":"1863d77a-baf2-4c78-bfe0-9b9b15bf319e","Type":"ContainerDied","Data":"e25a47ccc48ef788a21b4bde5fd99150988db99e81a954e338407a856052583d"}
Oct 30 01:30:39.699778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.698674    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s" event={"ID":"d62267d3-7fa0-436e-a6fb-bf197972e269","Type":"ContainerDied","Data":"4034807fd07ee285eced014d6ecf830cae911b8355943ddd41277795988a2da1"}
Oct 30 01:30:39.699778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.698884    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"
Oct 30 01:30:39.727703 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.727680    5104 scope.go:122] "RemoveContainer" containerID="1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506"
Oct 30 01:30:39.728049 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:39.728002    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506\": container with ID starting with 1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506 not found: ID does not exist" containerID="1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506"
Oct 30 01:30:39.728153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.728055    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506"} err="failed to get container status \"1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506\": rpc error: code = NotFound desc = could not find container \"1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506\": container with ID starting with 1e3aa2e8c468891929a11766ccf91ad5b8691d6489efcd01adb114225566e506 not found: ID does not exist"
Oct 30 01:30:39.728153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.728075    5104 scope.go:122] "RemoveContainer" containerID="110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f"
Oct 30 01:30:39.737236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.737208    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:30:39.746221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.746201    5104 scope.go:122] "RemoveContainer" containerID="110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f"
Oct 30 01:30:39.746601 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:39.746575    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f\": container with ID starting with 110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f not found: ID does not exist" containerID="110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f"
Oct 30 01:30:39.746673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.746610    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f"} err="failed to get container status \"110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f\": rpc error: code = NotFound desc = could not find container \"110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f\": container with ID starting with 110847d0579d8d1817c5aa0c690c854308552b5e6e595147b7689edab24c243f not found: ID does not exist"
Oct 30 01:30:39.746673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.746631    5104 scope.go:122] "RemoveContainer" containerID="470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78"
Oct 30 01:30:39.758571 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.758545    5104 scope.go:122] "RemoveContainer" containerID="470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78"
Oct 30 01:30:39.758971 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:39.758941    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78\": container with ID starting with 470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78 not found: ID does not exist" containerID="470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78"
Oct 30 01:30:39.759070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.758982    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78"} err="failed to get container status \"470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78\": rpc error: code = NotFound desc = could not find container \"470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78\": container with ID starting with 470313d39ce484b98c8777b68a71fdd24996a10e6cc51d574925153ded807f78 not found: ID does not exist"
Oct 30 01:30:39.759070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.759007    5104 scope.go:122] "RemoveContainer" containerID="039e32a6a5d40f1710f832cd421440b987a1792c49af3424b341cce0995dc5ce"
Oct 30 01:30:39.769552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.768912    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93218cf5-525b-4b93-8ef4-63c8ca95408d-kube-api-access-xnbc9\" (UniqueName: \"kubernetes.io/projected/93218cf5-525b-4b93-8ef4-63c8ca95408d-kube-api-access-xnbc9\") pod \"93218cf5-525b-4b93-8ef4-63c8ca95408d\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") "
Oct 30 01:30:39.769552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.768970    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-kube-api-access-s6ddr\" (UniqueName: \"kubernetes.io/projected/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-kube-api-access-s6ddr\") pod \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") "
Oct 30 01:30:39.769552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.769030    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-plugins-config-volume\") pod \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\" (UID: \"1863d77a-baf2-4c78-bfe0-9b9b15bf319e\") "
Oct 30 01:30:39.769552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.769064    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93218cf5-525b-4b93-8ef4-63c8ca95408d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93218cf5-525b-4b93-8ef4-63c8ca95408d-plugins-config-volume\") pod \"93218cf5-525b-4b93-8ef4-63c8ca95408d\" (UID: \"93218cf5-525b-4b93-8ef4-63c8ca95408d\") "
Oct 30 01:30:39.769906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.769875    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-plugins-config-volume" pod "1863d77a-baf2-4c78-bfe0-9b9b15bf319e" (UID: "1863d77a-baf2-4c78-bfe0-9b9b15bf319e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:39.769997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.769968    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93218cf5-525b-4b93-8ef4-63c8ca95408d-plugins-config-volume" pod "93218cf5-525b-4b93-8ef4-63c8ca95408d" (UID: "93218cf5-525b-4b93-8ef4-63c8ca95408d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:39.775937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.775898    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-kube-api-access-s6ddr" pod "1863d77a-baf2-4c78-bfe0-9b9b15bf319e" (UID: "1863d77a-baf2-4c78-bfe0-9b9b15bf319e"). InnerVolumeSpecName "kube-api-access-s6ddr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:39.776222 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.776183    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93218cf5-525b-4b93-8ef4-63c8ca95408d-kube-api-access-xnbc9" pod "93218cf5-525b-4b93-8ef4-63c8ca95408d" (UID: "93218cf5-525b-4b93-8ef4-63c8ca95408d"). InnerVolumeSpecName "kube-api-access-xnbc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:39.795882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.795844    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"]
Oct 30 01:30:39.802006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.801974    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaf753-7c7f7564dc-k499s"]
Oct 30 01:30:39.870813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.870782    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbc9\" (UniqueName: \"kubernetes.io/projected/93218cf5-525b-4b93-8ef4-63c8ca95408d-kube-api-access-xnbc9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.870813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.870810    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ddr\" (UniqueName: \"kubernetes.io/projected/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-kube-api-access-s6ddr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.870813 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.870821    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1863d77a-baf2-4c78-bfe0-9b9b15bf319e-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.871138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.870834    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93218cf5-525b-4b93-8ef4-63c8ca95408d-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:39.898048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.898010    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"]
Oct 30 01:30:39.913114 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.913069    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-79d9bf4b54-7sck9"]
Oct 30 01:30:39.983908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.983870    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"]
Oct 30 01:30:39.998416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:39.998382    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eaf753-8649859f-nml4p"]
Oct 30 01:30:40.068487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.068439    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"]
Oct 30 01:30:40.075878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.075845    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-b9f5594c8-kms8b"]
Oct 30 01:30:40.245923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.245892    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 01:30:40.255606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.255575    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"]
Oct 30 01:30:40.266221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.266188    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-29bb56-hgj8x"]
Oct 30 01:30:40.304572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304533    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973d2316-df2e-4774-b177-1b799d70f8e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/973d2316-df2e-4774-b177-1b799d70f8e9-istiod-ca-cert\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.304752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304661    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-credential-socket\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.304752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304700    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-istio-token\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-istio-token\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.304836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304771    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-kube-api-access-swz7f\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-kube-api-access-swz7f\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.304875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304832    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/973d2316-df2e-4774-b177-1b799d70f8e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/973d2316-df2e-4774-b177-1b799d70f8e9-istio-podinfo\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.304945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304901    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-data\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.305002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304949    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-certs\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.305002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304947    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-credential-socket" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:40.305002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304972    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973d2316-df2e-4774-b177-1b799d70f8e9-istiod-ca-cert" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:40.305002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.304993    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-socket\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.305224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.305038    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-envoy\") pod \"973d2316-df2e-4774-b177-1b799d70f8e9\" (UID: \"973d2316-df2e-4774-b177-1b799d70f8e9\") "
Oct 30 01:30:40.305291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.305260    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-data" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:40.305397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.305375    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-certs" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:40.305664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.305572    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-socket" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:40.307799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.307741    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/973d2316-df2e-4774-b177-1b799d70f8e9-istio-podinfo" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:30:40.308259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.308192    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-envoy" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:40.308259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.308249    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-istio-token" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:40.310641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.310620    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.310641 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.310644    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.310846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.310655    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.310846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.310665    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.310846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.310674    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/973d2316-df2e-4774-b177-1b799d70f8e9-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.310846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.310692    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-kube-api-access-swz7f" pod "973d2316-df2e-4774-b177-1b799d70f8e9" (UID: "973d2316-df2e-4774-b177-1b799d70f8e9"). InnerVolumeSpecName "kube-api-access-swz7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:40.411752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.411698    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/973d2316-df2e-4774-b177-1b799d70f8e9-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.411752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.411741    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.411752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.411751    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swz7f\" (UniqueName: \"kubernetes.io/projected/973d2316-df2e-4774-b177-1b799d70f8e9-kube-api-access-swz7f\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.412013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.411764    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/973d2316-df2e-4774-b177-1b799d70f8e9-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:40.578531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.578451    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1863d77a-baf2-4c78-bfe0-9b9b15bf319e" path="/var/lib/kubelet/pods/1863d77a-baf2-4c78-bfe0-9b9b15bf319e/volumes"
Oct 30 01:30:40.579030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.579014    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57db9115-3fdd-4345-b8ae-7eec4a1f12fa" path="/var/lib/kubelet/pods/57db9115-3fdd-4345-b8ae-7eec4a1f12fa/volumes"
Oct 30 01:30:40.579389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.579376    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6768328a-78ee-4d01-9e6c-df92c1c6a8d4" path="/var/lib/kubelet/pods/6768328a-78ee-4d01-9e6c-df92c1c6a8d4/volumes"
Oct 30 01:30:40.580124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.580105    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e9679b-3bbb-4c82-a88b-252093291d4d" path="/var/lib/kubelet/pods/b4e9679b-3bbb-4c82-a88b-252093291d4d/volumes"
Oct 30 01:30:40.580674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.580644    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62267d3-7fa0-436e-a6fb-bf197972e269" path="/var/lib/kubelet/pods/d62267d3-7fa0-436e-a6fb-bf197972e269/volumes"
Oct 30 01:30:40.581117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.581095    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de81acc9-6f4e-4d26-81ff-50652d037811" path="/var/lib/kubelet/pods/de81acc9-6f4e-4d26-81ff-50652d037811/volumes"
Oct 30 01:30:40.714283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.714244    5104 generic.go:356] "Generic (PLEG): container finished" podID="973d2316-df2e-4774-b177-1b799d70f8e9" containerID="1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651" exitCode=0
Oct 30 01:30:40.714752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.714302    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"
Oct 30 01:30:40.714752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.714327    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" event={"ID":"973d2316-df2e-4774-b177-1b799d70f8e9","Type":"ContainerDied","Data":"1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651"}
Oct 30 01:30:40.714752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.714392    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk" event={"ID":"973d2316-df2e-4774-b177-1b799d70f8e9","Type":"ContainerDied","Data":"cbe826444cde700f3e4234dc483f5afd80e16f8840f9bc328dcc256b6fbb95df"}
Oct 30 01:30:40.714752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.714422    5104 scope.go:122] "RemoveContainer" containerID="1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651"
Oct 30 01:30:40.717182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.717151    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"
Oct 30 01:30:40.717182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.717169    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2" event={"ID":"93218cf5-525b-4b93-8ef4-63c8ca95408d","Type":"ContainerDied","Data":"93a7706c7f8afddd07a96feb59a992e63fb8fd6a29a2be97c10a32d12bbf94cf"}
Oct 30 01:30:40.727231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.727207    5104 scope.go:122] "RemoveContainer" containerID="1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651"
Oct 30 01:30:40.727522 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:40.727502    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651\": container with ID starting with 1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651 not found: ID does not exist" containerID="1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651"
Oct 30 01:30:40.727567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.727531    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651"} err="failed to get container status \"1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651\": rpc error: code = NotFound desc = could not find container \"1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651\": container with ID starting with 1afe5134c90518ecf048f09f11e8d67595bb7df51267ea0fac6e3d14d7124651 not found: ID does not exist"
Oct 30 01:30:40.727567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.727556    5104 scope.go:122] "RemoveContainer" containerID="07e5c3a190c79a42a8203f601424914d8344f038d523b722809b6fed1be4c941"
Oct 30 01:30:40.774190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.774099    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"]
Oct 30 01:30:40.785264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.785226    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-kclfk"]
Oct 30 01:30:40.828697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.828660    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"]
Oct 30 01:30:40.840307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:40.840276    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-29bb56-6ddf6df5f5-fbkj2"]
Oct 30 01:30:42.616128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:42.616081    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93218cf5-525b-4b93-8ef4-63c8ca95408d" path="/var/lib/kubelet/pods/93218cf5-525b-4b93-8ef4-63c8ca95408d/volumes"
Oct 30 01:30:42.616856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:42.616821    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" path="/var/lib/kubelet/pods/973d2316-df2e-4774-b177-1b799d70f8e9/volumes"
Oct 30 01:30:45.011484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:45.007272    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"]
Oct 30 01:30:45.011484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:45.008889    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy" containerID="cri-o://7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f" gracePeriod=30
Oct 30 01:30:49.185652 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.185611    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"]
Oct 30 01:30:49.186236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186220    5104 cpu_manager.go:414] "containerMap: removing container" podUID="93218cf5-525b-4b93-8ef4-63c8ca95408d" containerName="payload-pre-processing"
Oct 30 01:30:49.186282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186237    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93218cf5-525b-4b93-8ef4-63c8ca95408d" containerName="payload-pre-processing"
Oct 30 01:30:49.186282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186256    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d62267d3-7fa0-436e-a6fb-bf197972e269" containerName="maas-api"
Oct 30 01:30:49.186282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186263    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62267d3-7fa0-436e-a6fb-bf197972e269" containerName="maas-api"
Oct 30 01:30:49.186282 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186278    5104 cpu_manager.go:414] "containerMap: removing container" podUID="36a06c24-ebaf-43d1-bc51-3046fb9d6c04" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186283    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a06c24-ebaf-43d1-bc51-3046fb9d6c04" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186297    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1863d77a-baf2-4c78-bfe0-9b9b15bf319e" containerName="payload-processing"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186302    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1863d77a-baf2-4c78-bfe0-9b9b15bf319e" containerName="payload-processing"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186313    5104 cpu_manager.go:414] "containerMap: removing container" podUID="430994cb-e380-4b7b-9f56-4bdec2053601" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186318    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430994cb-e380-4b7b-9f56-4bdec2053601" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186325    5104 cpu_manager.go:414] "containerMap: removing container" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186329    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186336    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6768328a-78ee-4d01-9e6c-df92c1c6a8d4" containerName="payload-processing"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186341    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6768328a-78ee-4d01-9e6c-df92c1c6a8d4" containerName="payload-processing"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186347    5104 cpu_manager.go:414] "containerMap: removing container" podUID="57db9115-3fdd-4345-b8ae-7eec4a1f12fa" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186352    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57db9115-3fdd-4345-b8ae-7eec4a1f12fa" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186359    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186364    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186375    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b4e9679b-3bbb-4c82-a88b-252093291d4d" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186379    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e9679b-3bbb-4c82-a88b-252093291d4d" containerName="revoke-keys"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186399    5104 cpu_manager.go:414] "containerMap: removing container" podUID="de81acc9-6f4e-4d26-81ff-50652d037811" containerName="payload-pre-processing"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186403    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de81acc9-6f4e-4d26-81ff-50652d037811" containerName="payload-pre-processing"
Oct 30 01:30:49.186493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186493    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="93218cf5-525b-4b93-8ef4-63c8ca95408d" containerName="payload-pre-processing"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186505    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62267d3-7fa0-436e-a6fb-bf197972e269" containerName="maas-api"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186515    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a06c24-ebaf-43d1-bc51-3046fb9d6c04" containerName="revoke-keys"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186522    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1863d77a-baf2-4c78-bfe0-9b9b15bf319e" containerName="payload-processing"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186527    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="430994cb-e380-4b7b-9f56-4bdec2053601" containerName="revoke-keys"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186533    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="973d2316-df2e-4774-b177-1b799d70f8e9" containerName="istio-proxy"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186538    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6768328a-78ee-4d01-9e6c-df92c1c6a8d4" containerName="payload-processing"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186543    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0b81d4-dd2c-4140-8ad3-e13a4da9d840" containerName="cleanup"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186548    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="57db9115-3fdd-4345-b8ae-7eec4a1f12fa" containerName="revoke-keys"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186553    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c35be6b-f3d3-4394-9e50-ff646a452e86" containerName="main"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186560    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e9679b-3bbb-4c82-a88b-252093291d4d" containerName="revoke-keys"
Oct 30 01:30:49.187086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.186568    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="de81acc9-6f4e-4d26-81ff-50652d037811" containerName="payload-pre-processing"
Oct 30 01:30:49.191582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.191561    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.211355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.211328    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:49.222340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.222310    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:30:49.269857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.269820    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8fn\" (UniqueName: \"kubernetes.io/projected/444f7cee-d088-490f-a354-2bbe74ab6be8-kube-api-access-4d8fn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.270067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.269930    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.270067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.270004    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.373037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.372988    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8fn\" (UniqueName: \"kubernetes.io/projected/444f7cee-d088-490f-a354-2bbe74ab6be8-kube-api-access-4d8fn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.373253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.373110    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.373253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.373201    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.374060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.374024    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.374517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.374489    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"]
Oct 30 01:30:49.376411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.376308    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.399403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.399367    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8fn\" (UniqueName: \"kubernetes.io/projected/444f7cee-d088-490f-a354-2bbe74ab6be8-kube-api-access-4d8fn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.507471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.507373    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:49.832391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:49.832361    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"]
Oct 30 01:30:49.834956 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:49.834925    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444f7cee_d088_490f_a354_2bbe74ab6be8.slice/crio-130dd2f1fd8742ea44e8a8c1446961c8e1f286d45c26c6c51be77c6030fdaf9d WatchSource:0}: Error finding container 130dd2f1fd8742ea44e8a8c1446961c8e1f286d45c26c6c51be77c6030fdaf9d: Status 404 returned error can't find the container with id 130dd2f1fd8742ea44e8a8c1446961c8e1f286d45c26c6c51be77c6030fdaf9d
Oct 30 01:30:50.302584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.302560    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:30:50.400301 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400250    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-socket\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400357    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-credential-socket\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400408    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istiod-ca-cert\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400432    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-kube-api-access-h6hzk\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-kube-api-access-h6hzk\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400496    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-token\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-token\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400581    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-data\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400605    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-envoy\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400632    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-certs\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400647    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-socket" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:50.400947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400662    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-podinfo\") pod \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\" (UID: \"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa\") "
Oct 30 01:30:50.400947 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.400861    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-credential-socket" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:50.401294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.401154    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-certs" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:50.401294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.401264    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-data" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:50.401484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.401363    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istiod-ca-cert" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:50.401484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.401418    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.401484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.401441    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.401484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.401472    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.401724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.401487    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.403259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.403236    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-kube-api-access-h6hzk" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "kube-api-access-h6hzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:50.408963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.408919    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-envoy" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:50.409079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.409040    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-podinfo" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:30:50.409544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.409520    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-token" pod "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" (UID: "97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:50.502846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.502803    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.502846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.502834    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.502846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.502845    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.502846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.502854    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6hzk\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-kube-api-access-h6hzk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.502846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.502863    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:50.889916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.889872    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d" event={"ID":"444f7cee-d088-490f-a354-2bbe74ab6be8","Type":"ContainerStarted","Data":"8e6ad3933e6055f0d925f5b038fe9c858607a978d62abab0175d15c37a18d2ed"}
Oct 30 01:30:50.890137 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.889946    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d" event={"ID":"444f7cee-d088-490f-a354-2bbe74ab6be8","Type":"ContainerStarted","Data":"130dd2f1fd8742ea44e8a8c1446961c8e1f286d45c26c6c51be77c6030fdaf9d"}
Oct 30 01:30:50.891845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.891817    5104 generic.go:356] "Generic (PLEG): container finished" podID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerID="7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f" exitCode=0
Oct 30 01:30:50.891980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.891849    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" event={"ID":"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa","Type":"ContainerDied","Data":"7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f"}
Oct 30 01:30:50.891980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.891868    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"
Oct 30 01:30:50.891980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.891904    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9" event={"ID":"97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa","Type":"ContainerDied","Data":"731b2dba109ae9d123c7fc174e01aad2d321461da071b70b2793454d2069f6f3"}
Oct 30 01:30:50.891980 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.891929    5104 scope.go:122] "RemoveContainer" containerID="7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f"
Oct 30 01:30:50.904134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.904111    5104 scope.go:122] "RemoveContainer" containerID="7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f"
Oct 30 01:30:50.904437 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:30:50.904398    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f\": container with ID starting with 7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f not found: ID does not exist" containerID="7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f"
Oct 30 01:30:50.904535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.904445    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f"} err="failed to get container status \"7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f\": rpc error: code = NotFound desc = could not find container \"7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f\": container with ID starting with 7547bf81d12d39f323ccde9002bbab367ed1329a95f19a14fb7b77568c38a63f not found: ID does not exist"
Oct 30 01:30:50.949512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:50.949447    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d" podStartSLOduration=1.949433062 podStartE2EDuration="1.949433062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:50.946228665 +0000 UTC m=+2892.493919168" watchObservedRunningTime="2026-10-30 01:30:50.949433062 +0000 UTC m=+2892.497123557"
Oct 30 01:30:51.008659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.008620    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"]
Oct 30 01:30:51.019065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.019022    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-trsx9"]
Oct 30 01:30:51.534247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.534211    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"]
Oct 30 01:30:51.534885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.534864    5104 cpu_manager.go:414] "containerMap: removing container" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy"
Oct 30 01:30:51.534963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.534887    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy"
Oct 30 01:30:51.535088 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.535073    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" containerName="istio-proxy"
Oct 30 01:30:51.538764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.538747    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.586181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.586152    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"]
Oct 30 01:30:51.617202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.617175    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.617373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.617302    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.617373 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.617335    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4c6\" (UniqueName: \"kubernetes.io/projected/c617c6e8-f91d-4107-80c2-df957da47170-kube-api-access-4f4c6\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.719245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.719204    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.719491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.719305    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.719491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.719333    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4c6\" (UniqueName: \"kubernetes.io/projected/c617c6e8-f91d-4107-80c2-df957da47170-kube-api-access-4f4c6\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.720144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.720115    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.721963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.721936    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.743188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.743163    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4c6\" (UniqueName: \"kubernetes.io/projected/c617c6e8-f91d-4107-80c2-df957da47170-kube-api-access-4f4c6\") pod \"maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.853554 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.853472    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:51.902050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.901753    5104 generic.go:356] "Generic (PLEG): container finished" podID="444f7cee-d088-490f-a354-2bbe74ab6be8" containerID="8e6ad3933e6055f0d925f5b038fe9c858607a978d62abab0175d15c37a18d2ed" exitCode=0
Oct 30 01:30:51.902050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:51.901795    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d" event={"ID":"444f7cee-d088-490f-a354-2bbe74ab6be8","Type":"ContainerDied","Data":"8e6ad3933e6055f0d925f5b038fe9c858607a978d62abab0175d15c37a18d2ed"}
Oct 30 01:30:52.182822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:52.182786    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"]
Oct 30 01:30:52.184396 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:30:52.184366    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc617c6e8_f91d_4107_80c2_df957da47170.slice/crio-bb15849560f5bc173e9d7872acb6e60a7b48b33707f12c53152ac18edc17f7cb WatchSource:0}: Error finding container bb15849560f5bc173e9d7872acb6e60a7b48b33707f12c53152ac18edc17f7cb: Status 404 returned error can't find the container with id bb15849560f5bc173e9d7872acb6e60a7b48b33707f12c53152ac18edc17f7cb
Oct 30 01:30:52.577245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:52.577217    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa" path="/var/lib/kubelet/pods/97a4cd1e-7da7-44a9-aae6-8d1e7e9064aa/volumes"
Oct 30 01:30:52.913863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:52.913765    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm" event={"ID":"c617c6e8-f91d-4107-80c2-df957da47170","Type":"ContainerStarted","Data":"c70d6b0cb52ee477d2c3cbefbc1681c54ae409b57dca0e9777d271260a171d09"}
Oct 30 01:30:52.913863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:52.913836    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm" event={"ID":"c617c6e8-f91d-4107-80c2-df957da47170","Type":"ContainerStarted","Data":"bb15849560f5bc173e9d7872acb6e60a7b48b33707f12c53152ac18edc17f7cb"}
Oct 30 01:30:52.968712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:52.968658    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm" podStartSLOduration=1.968641871 podStartE2EDuration="1.968641871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:52.967613273 +0000 UTC m=+2894.515303770" watchObservedRunningTime="2026-10-30 01:30:52.968641871 +0000 UTC m=+2894.516332366"
Oct 30 01:30:53.068677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.068649    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:53.074279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.074250    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"]
Oct 30 01:30:53.082525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.082498    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db92d77-fnbv7"]
Oct 30 01:30:53.168890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.168857    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444f7cee-d088-490f-a354-2bbe74ab6be8-kube-api-access-4d8fn\" (UniqueName: \"kubernetes.io/projected/444f7cee-d088-490f-a354-2bbe74ab6be8-kube-api-access-4d8fn\") pod \"444f7cee-d088-490f-a354-2bbe74ab6be8\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") "
Oct 30 01:30:53.169065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.168967    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-openshift-ca\") pod \"444f7cee-d088-490f-a354-2bbe74ab6be8\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") "
Oct 30 01:30:53.169065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.169030    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-xks-ca\") pod \"444f7cee-d088-490f-a354-2bbe74ab6be8\" (UID: \"444f7cee-d088-490f-a354-2bbe74ab6be8\") "
Oct 30 01:30:53.169735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.169697    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-openshift-ca" pod "444f7cee-d088-490f-a354-2bbe74ab6be8" (UID: "444f7cee-d088-490f-a354-2bbe74ab6be8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:53.172109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.172078    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-xks-ca" pod "444f7cee-d088-490f-a354-2bbe74ab6be8" (UID: "444f7cee-d088-490f-a354-2bbe74ab6be8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:53.172244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.172090    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444f7cee-d088-490f-a354-2bbe74ab6be8-kube-api-access-4d8fn" pod "444f7cee-d088-490f-a354-2bbe74ab6be8" (UID: "444f7cee-d088-490f-a354-2bbe74ab6be8"). InnerVolumeSpecName "kube-api-access-4d8fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:53.270699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.270663    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8fn\" (UniqueName: \"kubernetes.io/projected/444f7cee-d088-490f-a354-2bbe74ab6be8-kube-api-access-4d8fn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:53.270699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.270696    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:53.270699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.270707    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444f7cee-d088-490f-a354-2bbe74ab6be8-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:53.924427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.924397    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"
Oct 30 01:30:53.924867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.924431    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d" event={"ID":"444f7cee-d088-490f-a354-2bbe74ab6be8","Type":"ContainerDied","Data":"130dd2f1fd8742ea44e8a8c1446961c8e1f286d45c26c6c51be77c6030fdaf9d"}
Oct 30 01:30:53.924867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.924528    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="130dd2f1fd8742ea44e8a8c1446961c8e1f286d45c26c6c51be77c6030fdaf9d"
Oct 30 01:30:53.926676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.926650    5104 generic.go:356] "Generic (PLEG): container finished" podID="c617c6e8-f91d-4107-80c2-df957da47170" containerID="c70d6b0cb52ee477d2c3cbefbc1681c54ae409b57dca0e9777d271260a171d09" exitCode=0
Oct 30 01:30:53.926833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:53.926723    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm" event={"ID":"c617c6e8-f91d-4107-80c2-df957da47170","Type":"ContainerDied","Data":"c70d6b0cb52ee477d2c3cbefbc1681c54ae409b57dca0e9777d271260a171d09"}
Oct 30 01:30:54.082482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:54.082098    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"]
Oct 30 01:30:54.082482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:54.082342    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy" containerID="cri-o://37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa" gracePeriod=30
Oct 30 01:30:54.578016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:54.577981    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3370d60f-e20d-4420-b2cb-32d15d5e3e59" path="/var/lib/kubelet/pods/3370d60f-e20d-4420-b2cb-32d15d5e3e59/volumes"
Oct 30 01:30:54.785969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:54.785934    5104 scope.go:122] "RemoveContainer" containerID="56e2d6d212decd59f06871d14b9793a031565b95c5b08b5f1ea5d5017dcfba0f"
Oct 30 01:30:54.798177 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:54.798155    5104 scope.go:122] "RemoveContainer" containerID="8f27a8d8e066d69b5183afb38aa06727ead3d61484f8845d094f11e9ebad78c6"
Oct 30 01:30:55.123145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.123114    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:55.297170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.297135    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-xks-ca\") pod \"c617c6e8-f91d-4107-80c2-df957da47170\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") "
Oct 30 01:30:55.297326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.297241    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c617c6e8-f91d-4107-80c2-df957da47170-kube-api-access-4f4c6\" (UniqueName: \"kubernetes.io/projected/c617c6e8-f91d-4107-80c2-df957da47170-kube-api-access-4f4c6\") pod \"c617c6e8-f91d-4107-80c2-df957da47170\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") "
Oct 30 01:30:55.297490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.297470    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-openshift-ca\") pod \"c617c6e8-f91d-4107-80c2-df957da47170\" (UID: \"c617c6e8-f91d-4107-80c2-df957da47170\") "
Oct 30 01:30:55.297967 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.297935    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-openshift-ca" pod "c617c6e8-f91d-4107-80c2-df957da47170" (UID: "c617c6e8-f91d-4107-80c2-df957da47170"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:55.299307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.299276    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-xks-ca" pod "c617c6e8-f91d-4107-80c2-df957da47170" (UID: "c617c6e8-f91d-4107-80c2-df957da47170"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:55.299629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.299608    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c617c6e8-f91d-4107-80c2-df957da47170-kube-api-access-4f4c6" pod "c617c6e8-f91d-4107-80c2-df957da47170" (UID: "c617c6e8-f91d-4107-80c2-df957da47170"). InnerVolumeSpecName "kube-api-access-4f4c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:55.370129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.370045    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:30:55.400370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.400303    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:55.400370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.400360    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c617c6e8-f91d-4107-80c2-df957da47170-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:55.400370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.400377    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4c6\" (UniqueName: \"kubernetes.io/projected/c617c6e8-f91d-4107-80c2-df957da47170-kube-api-access-4f4c6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:55.944614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.944584    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"
Oct 30 01:30:55.944796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.944610    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm" event={"ID":"c617c6e8-f91d-4107-80c2-df957da47170","Type":"ContainerDied","Data":"bb15849560f5bc173e9d7872acb6e60a7b48b33707f12c53152ac18edc17f7cb"}
Oct 30 01:30:55.944796 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:55.944668    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb15849560f5bc173e9d7872acb6e60a7b48b33707f12c53152ac18edc17f7cb"
Oct 30 01:30:59.355965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.355898    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 01:30:59.460113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460078    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-podinfo\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460138    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-socket\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460225    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-envoy\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460278    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-kube-api-access-6czx5\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-kube-api-access-6czx5\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460308    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-certs\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460357    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65623efa-d0c0-455c-a784-3b48e1e9eb24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65623efa-d0c0-455c-a784-3b48e1e9eb24-istiod-ca-cert\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460394    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-token\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-token\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460433    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-data\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460504    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-socket" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.460618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460535    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-credential-socket\") pod \"65623efa-d0c0-455c-a784-3b48e1e9eb24\" (UID: \"65623efa-d0c0-455c-a784-3b48e1e9eb24\") "
Oct 30 01:30:59.460923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.460846    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65623efa-d0c0-455c-a784-3b48e1e9eb24-istiod-ca-cert" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:59.461248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.461108    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-certs" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.461248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.461137    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65623efa-d0c0-455c-a784-3b48e1e9eb24-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.461248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.461159    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.461248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.461206    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-data" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.461248 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.461216    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-credential-socket" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.462913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.462874    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-kube-api-access-6czx5" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "kube-api-access-6czx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:59.463020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.462976    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-envoy" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.463212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.463195    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-podinfo" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:30:59.463346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.463327    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-token" pod "65623efa-d0c0-455c-a784-3b48e1e9eb24" (UID: "65623efa-d0c0-455c-a784-3b48e1e9eb24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:59.562341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.562302    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.562341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.562334    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.562341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.562345    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.562659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.562355    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.562659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.562366    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.562659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.562377    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65623efa-d0c0-455c-a784-3b48e1e9eb24-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.562659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.562386    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czx5\" (UniqueName: \"kubernetes.io/projected/65623efa-d0c0-455c-a784-3b48e1e9eb24-kube-api-access-6czx5\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.983740 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.983674    5104 generic.go:356] "Generic (PLEG): container finished" podID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerID="37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa" exitCode=0
Oct 30 01:30:59.983979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.983742    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"
Oct 30 01:30:59.984063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.984034    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" event={"ID":"65623efa-d0c0-455c-a784-3b48e1e9eb24","Type":"ContainerDied","Data":"37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa"}
Oct 30 01:30:59.984151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.984092    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf" event={"ID":"65623efa-d0c0-455c-a784-3b48e1e9eb24","Type":"ContainerDied","Data":"4b8e1a23da529ec6fc242c445e919d5cee3f53f5580ea8a7fe2dce5580ec5c83"}
Oct 30 01:30:59.984151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.984125    5104 scope.go:122] "RemoveContainer" containerID="37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa"
Oct 30 01:30:59.997262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:30:59.997235    5104 scope.go:122] "RemoveContainer" containerID="c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a"
Oct 30 01:31:00.008234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:00.008209    5104 scope.go:122] "RemoveContainer" containerID="37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa"
Oct 30 01:31:00.008595 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:00.008573    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa\": container with ID starting with 37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa not found: ID does not exist" containerID="37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa"
Oct 30 01:31:00.008689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:00.008604    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa"} err="failed to get container status \"37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa\": rpc error: code = NotFound desc = could not find container \"37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa\": container with ID starting with 37b9b876aaa7dc43d9197db27eb85ec8337f1e86a50e44e8762fcdd76ff8e6aa not found: ID does not exist"
Oct 30 01:31:00.008689 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:00.008623    5104 scope.go:122] "RemoveContainer" containerID="c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a"
Oct 30 01:31:00.008897 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:00.008880    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a\": container with ID starting with c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a not found: ID does not exist" containerID="c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a"
Oct 30 01:31:00.008954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:00.008904    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a"} err="failed to get container status \"c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a\": rpc error: code = NotFound desc = could not find container \"c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a\": container with ID starting with c5bae8212f140a53830957e010b23f65a9fdff71008b3f8ed27e1a94f1c4756a not found: ID does not exist"
Oct 30 01:31:00.045836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:00.045803    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"]
Oct 30 01:31:00.057218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:00.057177    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-dkbkf"]
Oct 30 01:31:00.614762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:00.614721    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" path="/var/lib/kubelet/pods/65623efa-d0c0-455c-a784-3b48e1e9eb24/volumes"
Oct 30 01:31:03.044787 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:03.044746    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"]
Oct 30 01:31:03.056082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:03.056046    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349b4982-trjdx"]
Oct 30 01:31:04.576874 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:04.576840    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f74809-f096-4414-8e9d-c6fab705fe3d" path="/var/lib/kubelet/pods/57f74809-f096-4414-8e9d-c6fab705fe3d/volumes"
Oct 30 01:31:08.171091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:08.171054    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"]
Oct 30 01:31:08.181146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:08.181045    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e4f8a6-7rl8d"]
Oct 30 01:31:08.579203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:08.579114    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444f7cee-d088-490f-a354-2bbe74ab6be8" path="/var/lib/kubelet/pods/444f7cee-d088-490f-a354-2bbe74ab6be8/volumes"
Oct 30 01:31:09.105146 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:09.105113    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:31:14.695516 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:14.683050    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:31:15.391670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.391633    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"]
Oct 30 01:31:15.391895 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.391843    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm" podUID="df745373-af7c-4f1f-8ada-df75bb8de3ad" containerName="maas-api" containerID="cri-o://6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81" gracePeriod=30
Oct 30 01:31:15.607249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.607210    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"]
Oct 30 01:31:15.607556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.607512    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm" podUID="9a8fe731-d987-4488-9c4e-ff7a5090d781" containerName="payload-processing" containerID="cri-o://f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef" gracePeriod=30
Oct 30 01:31:15.632782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.631994    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"]
Oct 30 01:31:15.632782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.632218    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794" podUID="a18ad5ff-2880-4e4f-bc12-5690e75e570a" containerName="payload-pre-processing" containerID="cri-o://2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62" gracePeriod=30
Oct 30 01:31:15.768165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.768133    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:31:15.880397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.880360    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df745373-af7c-4f1f-8ada-df75bb8de3ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df745373-af7c-4f1f-8ada-df75bb8de3ad-openshift-service-ca\") pod \"df745373-af7c-4f1f-8ada-df75bb8de3ad\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") "
Oct 30 01:31:15.880647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.880519    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-metrics-certs\") pod \"df745373-af7c-4f1f-8ada-df75bb8de3ad\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") "
Oct 30 01:31:15.880647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.880602    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df745373-af7c-4f1f-8ada-df75bb8de3ad-kube-api-access-6tt95\" (UniqueName: \"kubernetes.io/projected/df745373-af7c-4f1f-8ada-df75bb8de3ad-kube-api-access-6tt95\") pod \"df745373-af7c-4f1f-8ada-df75bb8de3ad\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") "
Oct 30 01:31:15.880799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.880660    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-maas-api-tls\") pod \"df745373-af7c-4f1f-8ada-df75bb8de3ad\" (UID: \"df745373-af7c-4f1f-8ada-df75bb8de3ad\") "
Oct 30 01:31:15.881305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.881227    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df745373-af7c-4f1f-8ada-df75bb8de3ad-openshift-service-ca" pod "df745373-af7c-4f1f-8ada-df75bb8de3ad" (UID: "df745373-af7c-4f1f-8ada-df75bb8de3ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:15.888448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.888416    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-metrics-certs" pod "df745373-af7c-4f1f-8ada-df75bb8de3ad" (UID: "df745373-af7c-4f1f-8ada-df75bb8de3ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:15.895185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.893658    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-maas-api-tls" pod "df745373-af7c-4f1f-8ada-df75bb8de3ad" (UID: "df745373-af7c-4f1f-8ada-df75bb8de3ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:15.895185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.893687    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df745373-af7c-4f1f-8ada-df75bb8de3ad-kube-api-access-6tt95" pod "df745373-af7c-4f1f-8ada-df75bb8de3ad" (UID: "df745373-af7c-4f1f-8ada-df75bb8de3ad"). InnerVolumeSpecName "kube-api-access-6tt95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:15.909697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:15.909667    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:31:16.033253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.027223    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8fe731-d987-4488-9c4e-ff7a5090d781-kube-api-access-l46hb\" (UniqueName: \"kubernetes.io/projected/9a8fe731-d987-4488-9c4e-ff7a5090d781-kube-api-access-l46hb\") pod \"9a8fe731-d987-4488-9c4e-ff7a5090d781\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") "
Oct 30 01:31:16.033253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.027275    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8fe731-d987-4488-9c4e-ff7a5090d781-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8fe731-d987-4488-9c4e-ff7a5090d781-plugins-config-volume\") pod \"9a8fe731-d987-4488-9c4e-ff7a5090d781\" (UID: \"9a8fe731-d987-4488-9c4e-ff7a5090d781\") "
Oct 30 01:31:16.033253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.027628    5104 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df745373-af7c-4f1f-8ada-df75bb8de3ad-openshift-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.033253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.027653    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.033253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.027668    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tt95\" (UniqueName: \"kubernetes.io/projected/df745373-af7c-4f1f-8ada-df75bb8de3ad-kube-api-access-6tt95\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.033253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.027682    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df745373-af7c-4f1f-8ada-df75bb8de3ad-maas-api-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.033253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.031310    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8fe731-d987-4488-9c4e-ff7a5090d781-plugins-config-volume" pod "9a8fe731-d987-4488-9c4e-ff7a5090d781" (UID: "9a8fe731-d987-4488-9c4e-ff7a5090d781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:16.038477 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.038426    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8fe731-d987-4488-9c4e-ff7a5090d781-kube-api-access-l46hb" pod "9a8fe731-d987-4488-9c4e-ff7a5090d781" (UID: "9a8fe731-d987-4488-9c4e-ff7a5090d781"). InnerVolumeSpecName "kube-api-access-l46hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:16.048680 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.048650    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:31:16.108045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.108007    5104 generic.go:356] "Generic (PLEG): container finished" podID="9a8fe731-d987-4488-9c4e-ff7a5090d781" containerID="f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef" exitCode=0
Oct 30 01:31:16.108256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.108060    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"
Oct 30 01:31:16.108256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.108086    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm" event={"ID":"9a8fe731-d987-4488-9c4e-ff7a5090d781","Type":"ContainerDied","Data":"f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef"}
Oct 30 01:31:16.108256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.108155    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm" event={"ID":"9a8fe731-d987-4488-9c4e-ff7a5090d781","Type":"ContainerDied","Data":"27e4de0af9fb917e75dc1c050a52c524717ee8234c39e5505db772bb8673e99e"}
Oct 30 01:31:16.108256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.108181    5104 scope.go:122] "RemoveContainer" containerID="f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef"
Oct 30 01:31:16.116835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.116785    5104 generic.go:356] "Generic (PLEG): container finished" podID="df745373-af7c-4f1f-8ada-df75bb8de3ad" containerID="6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81" exitCode=0
Oct 30 01:31:16.116998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.116930    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm" event={"ID":"df745373-af7c-4f1f-8ada-df75bb8de3ad","Type":"ContainerDied","Data":"6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81"}
Oct 30 01:31:16.116998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.116984    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm" event={"ID":"df745373-af7c-4f1f-8ada-df75bb8de3ad","Type":"ContainerDied","Data":"f122c3c86a21dd6396a4f7b03d323dabf2384c8a6eda2eb6ae3274a03d696bc0"}
Oct 30 01:31:16.117368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.117344    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"
Oct 30 01:31:16.119958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.119923    5104 generic.go:356] "Generic (PLEG): container finished" podID="a18ad5ff-2880-4e4f-bc12-5690e75e570a" containerID="2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62" exitCode=0
Oct 30 01:31:16.120122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.120024    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794" event={"ID":"a18ad5ff-2880-4e4f-bc12-5690e75e570a","Type":"ContainerDied","Data":"2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62"}
Oct 30 01:31:16.120122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.120069    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794" event={"ID":"a18ad5ff-2880-4e4f-bc12-5690e75e570a","Type":"ContainerDied","Data":"a6ed0ceefd6bd711df16df35ac6ba2f8ae03d8f8742e605cb783c10b8426f68b"}
Oct 30 01:31:16.120271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.120154    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"
Oct 30 01:31:16.120943 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.120925    5104 scope.go:122] "RemoveContainer" containerID="f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef"
Oct 30 01:31:16.121541 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:16.121508    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef\": container with ID starting with f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef not found: ID does not exist" containerID="f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef"
Oct 30 01:31:16.121633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.121552    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef"} err="failed to get container status \"f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef\": rpc error: code = NotFound desc = could not find container \"f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef\": container with ID starting with f14b0c41d3292929da8ad09989e274e3d5f1d092f368f2684657450110bff4ef not found: ID does not exist"
Oct 30 01:31:16.121633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.121577    5104 scope.go:122] "RemoveContainer" containerID="6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81"
Oct 30 01:31:16.128827 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.128797    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18ad5ff-2880-4e4f-bc12-5690e75e570a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ad5ff-2880-4e4f-bc12-5690e75e570a-plugins-config-volume\") pod \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") "
Oct 30 01:31:16.129729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.128936    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18ad5ff-2880-4e4f-bc12-5690e75e570a-kube-api-access-jdlrp\" (UniqueName: \"kubernetes.io/projected/a18ad5ff-2880-4e4f-bc12-5690e75e570a-kube-api-access-jdlrp\") pod \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\" (UID: \"a18ad5ff-2880-4e4f-bc12-5690e75e570a\") "
Oct 30 01:31:16.129729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.129278    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18ad5ff-2880-4e4f-bc12-5690e75e570a-plugins-config-volume" pod "a18ad5ff-2880-4e4f-bc12-5690e75e570a" (UID: "a18ad5ff-2880-4e4f-bc12-5690e75e570a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:16.129729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.129595    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18ad5ff-2880-4e4f-bc12-5690e75e570a-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.129729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.129646    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46hb\" (UniqueName: \"kubernetes.io/projected/9a8fe731-d987-4488-9c4e-ff7a5090d781-kube-api-access-l46hb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.129729 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.129663    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8fe731-d987-4488-9c4e-ff7a5090d781-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.139039 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.138952    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18ad5ff-2880-4e4f-bc12-5690e75e570a-kube-api-access-jdlrp" pod "a18ad5ff-2880-4e4f-bc12-5690e75e570a" (UID: "a18ad5ff-2880-4e4f-bc12-5690e75e570a"). InnerVolumeSpecName "kube-api-access-jdlrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:16.139136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.139116    5104 scope.go:122] "RemoveContainer" containerID="6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81"
Oct 30 01:31:16.139532 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:16.139484    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81\": container with ID starting with 6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81 not found: ID does not exist" containerID="6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81"
Oct 30 01:31:16.139532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.139520    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81"} err="failed to get container status \"6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81\": rpc error: code = NotFound desc = could not find container \"6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81\": container with ID starting with 6de9cdf0a6b0cd1456246c998832230e589820a373a21b98273c15fe17124f81 not found: ID does not exist"
Oct 30 01:31:16.139686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.139541    5104 scope.go:122] "RemoveContainer" containerID="2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62"
Oct 30 01:31:16.152654 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.152616    5104 scope.go:122] "RemoveContainer" containerID="2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62"
Oct 30 01:31:16.152999 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:16.152971    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62\": container with ID starting with 2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62 not found: ID does not exist" containerID="2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62"
Oct 30 01:31:16.153081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.153013    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62"} err="failed to get container status \"2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62\": rpc error: code = NotFound desc = could not find container \"2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62\": container with ID starting with 2e0a85ff518873b19d55d52f569ecfcce51cfc21ae5d3f0154694ba0a9bfdb62 not found: ID does not exist"
Oct 30 01:31:16.193779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.193744    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"]
Oct 30 01:31:16.198921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.198889    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-679bd2-658f485b5b-fbjjm"]
Oct 30 01:31:16.232030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.231992    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlrp\" (UniqueName: \"kubernetes.io/projected/a18ad5ff-2880-4e4f-bc12-5690e75e570a-kube-api-access-jdlrp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:16.250528 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.248633    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"]
Oct 30 01:31:16.252390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.252359    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-679bd2-5klgm"]
Oct 30 01:31:16.255581 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.255549    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"]
Oct 30 01:31:16.272756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.272624    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-679bd2-69fbdcfd8d-z59zm"]
Oct 30 01:31:16.465665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.465631    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"]
Oct 30 01:31:16.469982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.469951    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-679bd2-77cfb56cf-kr794"]
Oct 30 01:31:16.576804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.576722    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8fe731-d987-4488-9c4e-ff7a5090d781" path="/var/lib/kubelet/pods/9a8fe731-d987-4488-9c4e-ff7a5090d781/volumes"
Oct 30 01:31:16.577151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.577137    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18ad5ff-2880-4e4f-bc12-5690e75e570a" path="/var/lib/kubelet/pods/a18ad5ff-2880-4e4f-bc12-5690e75e570a/volumes"
Oct 30 01:31:16.577568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.577554    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c617c6e8-f91d-4107-80c2-df957da47170" path="/var/lib/kubelet/pods/c617c6e8-f91d-4107-80c2-df957da47170/volumes"
Oct 30 01:31:16.578272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:16.578257    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df745373-af7c-4f1f-8ada-df75bb8de3ad" path="/var/lib/kubelet/pods/df745373-af7c-4f1f-8ada-df75bb8de3ad/volumes"
Oct 30 01:31:17.848564 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:17.848528    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"]
Oct 30 01:31:17.849201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:17.849144    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy" containerID="cri-o://826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f" gracePeriod=30
Oct 30 01:31:18.349116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349078    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"]
Oct 30 01:31:18.349908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349887    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c617c6e8-f91d-4107-80c2-df957da47170" containerName="revoke-keys"
Oct 30 01:31:18.349971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349911    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c617c6e8-f91d-4107-80c2-df957da47170" containerName="revoke-keys"
Oct 30 01:31:18.349971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349940    5104 cpu_manager.go:414] "containerMap: removing container" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy"
Oct 30 01:31:18.349971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349948    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy"
Oct 30 01:31:18.349971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349961    5104 cpu_manager.go:414] "containerMap: removing container" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy"
Oct 30 01:31:18.349971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349969    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349984    5104 cpu_manager.go:414] "containerMap: removing container" podUID="df745373-af7c-4f1f-8ada-df75bb8de3ad" containerName="maas-api"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.349992    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df745373-af7c-4f1f-8ada-df75bb8de3ad" containerName="maas-api"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350012    5104 cpu_manager.go:414] "containerMap: removing container" podUID="9a8fe731-d987-4488-9c4e-ff7a5090d781" containerName="payload-processing"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350019    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8fe731-d987-4488-9c4e-ff7a5090d781" containerName="payload-processing"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350037    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a18ad5ff-2880-4e4f-bc12-5690e75e570a" containerName="payload-pre-processing"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350043    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18ad5ff-2880-4e4f-bc12-5690e75e570a" containerName="payload-pre-processing"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350081    5104 cpu_manager.go:414] "containerMap: removing container" podUID="444f7cee-d088-490f-a354-2bbe74ab6be8" containerName="revoke-keys"
Oct 30 01:31:18.350148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350088    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444f7cee-d088-490f-a354-2bbe74ab6be8" containerName="revoke-keys"
Oct 30 01:31:18.350413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350213    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy"
Oct 30 01:31:18.350413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350227    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="65623efa-d0c0-455c-a784-3b48e1e9eb24" containerName="istio-proxy"
Oct 30 01:31:18.350413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350241    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="df745373-af7c-4f1f-8ada-df75bb8de3ad" containerName="maas-api"
Oct 30 01:31:18.350413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350251    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8fe731-d987-4488-9c4e-ff7a5090d781" containerName="payload-processing"
Oct 30 01:31:18.350413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350263    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18ad5ff-2880-4e4f-bc12-5690e75e570a" containerName="payload-pre-processing"
Oct 30 01:31:18.350413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350277    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="444f7cee-d088-490f-a354-2bbe74ab6be8" containerName="revoke-keys"
Oct 30 01:31:18.350413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.350287    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c617c6e8-f91d-4107-80c2-df957da47170" containerName="revoke-keys"
Oct 30 01:31:18.355800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.355774    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.375480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.374619    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:31:18.375480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.374687    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:18.378176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.378140    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"]
Oct 30 01:31:18.385115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.385079    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:31:18.385115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.385098    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:31:18.468572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.468529    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.468767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.468609    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.468767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.468759    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49jm\" (UniqueName: \"kubernetes.io/projected/f83a401d-3694-44e7-8ad5-04bd3940b107-kube-api-access-j49jm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.569660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.569622    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.569885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.569670    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.569885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.569699    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49jm\" (UniqueName: \"kubernetes.io/projected/f83a401d-3694-44e7-8ad5-04bd3940b107-kube-api-access-j49jm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.570370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.570342    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.572313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.572284    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.593539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.593506    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49jm\" (UniqueName: \"kubernetes.io/projected/f83a401d-3694-44e7-8ad5-04bd3940b107-kube-api-access-j49jm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.671231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.671200    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:18.987682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:18.987649    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"]
Oct 30 01:31:18.988908 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:31:18.988873    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83a401d_3694_44e7_8ad5_04bd3940b107.slice/crio-a40899fb497a09169f39c22c7f41c64e994e93842b2edb8728096a3c9f62004a WatchSource:0}: Error finding container a40899fb497a09169f39c22c7f41c64e994e93842b2edb8728096a3c9f62004a: Status 404 returned error can't find the container with id a40899fb497a09169f39c22c7f41c64e994e93842b2edb8728096a3c9f62004a
Oct 30 01:31:19.147345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:19.147304    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg" event={"ID":"f83a401d-3694-44e7-8ad5-04bd3940b107","Type":"ContainerStarted","Data":"a40899fb497a09169f39c22c7f41c64e994e93842b2edb8728096a3c9f62004a"}
Oct 30 01:31:20.157911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:20.157820    5104 generic.go:356] "Generic (PLEG): container finished" podID="f83a401d-3694-44e7-8ad5-04bd3940b107" containerID="5082cfc098d752e2062f9a1b87d218d2345038f0c559732c9303ffcaa05c2a78" exitCode=0
Oct 30 01:31:20.158300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:20.157912    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg" event={"ID":"f83a401d-3694-44e7-8ad5-04bd3940b107","Type":"ContainerDied","Data":"5082cfc098d752e2062f9a1b87d218d2345038f0c559732c9303ffcaa05c2a78"}
Oct 30 01:31:21.347256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.347228    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:21.509376 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.509337    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-openshift-ca\") pod \"f83a401d-3694-44e7-8ad5-04bd3940b107\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") "
Oct 30 01:31:21.509627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.509502    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83a401d-3694-44e7-8ad5-04bd3940b107-kube-api-access-j49jm\" (UniqueName: \"kubernetes.io/projected/f83a401d-3694-44e7-8ad5-04bd3940b107-kube-api-access-j49jm\") pod \"f83a401d-3694-44e7-8ad5-04bd3940b107\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") "
Oct 30 01:31:21.509627 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.509538    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-xks-ca\") pod \"f83a401d-3694-44e7-8ad5-04bd3940b107\" (UID: \"f83a401d-3694-44e7-8ad5-04bd3940b107\") "
Oct 30 01:31:21.509841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.509815    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-openshift-ca" pod "f83a401d-3694-44e7-8ad5-04bd3940b107" (UID: "f83a401d-3694-44e7-8ad5-04bd3940b107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:21.510043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.510014    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:21.511910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.511880    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-xks-ca" pod "f83a401d-3694-44e7-8ad5-04bd3940b107" (UID: "f83a401d-3694-44e7-8ad5-04bd3940b107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:21.512016 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.511927    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83a401d-3694-44e7-8ad5-04bd3940b107-kube-api-access-j49jm" pod "f83a401d-3694-44e7-8ad5-04bd3940b107" (UID: "f83a401d-3694-44e7-8ad5-04bd3940b107"). InnerVolumeSpecName "kube-api-access-j49jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:21.619008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.618921    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83a401d-3694-44e7-8ad5-04bd3940b107-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:21.619008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:21.618969    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49jm\" (UniqueName: \"kubernetes.io/projected/f83a401d-3694-44e7-8ad5-04bd3940b107-kube-api-access-j49jm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:22.176060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:22.176028    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"
Oct 30 01:31:22.176264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:22.176024    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg" event={"ID":"f83a401d-3694-44e7-8ad5-04bd3940b107","Type":"ContainerDied","Data":"a40899fb497a09169f39c22c7f41c64e994e93842b2edb8728096a3c9f62004a"}
Oct 30 01:31:22.176264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:22.176184    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40899fb497a09169f39c22c7f41c64e994e93842b2edb8728096a3c9f62004a"
Oct 30 01:31:23.158037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.158004    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:31:23.231522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.231475    5104 generic.go:356] "Generic (PLEG): container finished" podID="2c066f1b-c608-42af-800c-7167e4c48a80" containerID="826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f" exitCode=0
Oct 30 01:31:23.231722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.231526    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" event={"ID":"2c066f1b-c608-42af-800c-7167e4c48a80","Type":"ContainerDied","Data":"826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f"}
Oct 30 01:31:23.231722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.231585    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"
Oct 30 01:31:23.231722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.231613    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp" event={"ID":"2c066f1b-c608-42af-800c-7167e4c48a80","Type":"ContainerDied","Data":"07906b3dd3c47892e3e3b3de86adfff22400358379d7b7569eb50dab1543e6a9"}
Oct 30 01:31:23.231722 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.231636    5104 scope.go:122] "RemoveContainer" containerID="826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f"
Oct 30 01:31:23.252372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.252343    5104 scope.go:122] "RemoveContainer" containerID="826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f"
Oct 30 01:31:23.252811 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:23.252779    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f\": container with ID starting with 826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f not found: ID does not exist" containerID="826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f"
Oct 30 01:31:23.252940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.252822    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f"} err="failed to get container status \"826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f\": rpc error: code = NotFound desc = could not find container \"826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f\": container with ID starting with 826fda80574d86c8a1713ddb42578557f16fee9d6f21349c6c8d66e1d1df111f not found: ID does not exist"
Oct 30 01:31:23.341131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341096    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-envoy\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341160    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-credential-socket\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341220    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-istio-token\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-istio-token\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341249    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-socket\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341277    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c066f1b-c608-42af-800c-7167e4c48a80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c066f1b-c608-42af-800c-7167e4c48a80-istio-podinfo\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341303    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-certs\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341382    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-data\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341449    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c066f1b-c608-42af-800c-7167e4c48a80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c066f1b-c608-42af-800c-7167e4c48a80-istiod-ca-cert\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341514    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-kube-api-access-gvwcm\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-kube-api-access-gvwcm\") pod \"2c066f1b-c608-42af-800c-7167e4c48a80\" (UID: \"2c066f1b-c608-42af-800c-7167e4c48a80\") "
Oct 30 01:31:23.341673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341578    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-credential-socket" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:23.341673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341630    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-socket" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:23.341673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341659    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-certs" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:23.342005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341818    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-data" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:23.342005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.341981    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c066f1b-c608-42af-800c-7167e4c48a80-istiod-ca-cert" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:23.342151 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.342132    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.342220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.342158    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.342220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.342176    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.342220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.342191    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.342220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.342205    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c066f1b-c608-42af-800c-7167e4c48a80-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.343548 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.343519    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-envoy" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:23.343667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.343560    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c066f1b-c608-42af-800c-7167e4c48a80-istio-podinfo" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:31:23.343716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.343704    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-kube-api-access-gvwcm" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "kube-api-access-gvwcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:23.343767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.343743    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-istio-token" pod "2c066f1b-c608-42af-800c-7167e4c48a80" (UID: "2c066f1b-c608-42af-800c-7167e4c48a80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:23.443041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.442992    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvwcm\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-kube-api-access-gvwcm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.443041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.443028    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c066f1b-c608-42af-800c-7167e4c48a80-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.443041 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.443042    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c066f1b-c608-42af-800c-7167e4c48a80-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.443324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.443054    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c066f1b-c608-42af-800c-7167e4c48a80-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:23.450871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.450821    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"]
Oct 30 01:31:23.451194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.451155    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" containerID="cri-o://3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c" gracePeriod=30
Oct 30 01:31:23.596524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.591628    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"]
Oct 30 01:31:23.626856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.626813    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-p7szp"]
Oct 30 01:31:23.940001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.939965    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"]
Oct 30 01:31:23.940553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.940537    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy"
Oct 30 01:31:23.940624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.940555    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy"
Oct 30 01:31:23.940624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.940591    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f83a401d-3694-44e7-8ad5-04bd3940b107" containerName="revoke-keys"
Oct 30 01:31:23.940624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.940596    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83a401d-3694-44e7-8ad5-04bd3940b107" containerName="revoke-keys"
Oct 30 01:31:23.940734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.940667    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83a401d-3694-44e7-8ad5-04bd3940b107" containerName="revoke-keys"
Oct 30 01:31:23.940734 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.940684    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" containerName="istio-proxy"
Oct 30 01:31:23.945570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.945546    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:23.947253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.947226    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:23.947408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.947391    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:23.947452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.947421    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn58b\" (UniqueName: \"kubernetes.io/projected/50c2de21-e6a2-4075-838d-562a35d48252-kube-api-access-fn58b\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:23.959800 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.959777    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:31:23.959960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.959865    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:31:23.960917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.960893    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:23.961156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.961136    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:31:23.963357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:23.963331    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"]
Oct 30 01:31:24.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.048020    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:24.048070 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.048070    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn58b\" (UniqueName: \"kubernetes.io/projected/50c2de21-e6a2-4075-838d-562a35d48252-kube-api-access-fn58b\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:24.048359 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.048167    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:24.049026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.049000    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:24.050549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.050525    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:24.072392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.072361    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn58b\" (UniqueName: \"kubernetes.io/projected/50c2de21-e6a2-4075-838d-562a35d48252-kube-api-access-fn58b\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:24.260024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.259932    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:24.577775 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:31:24.574644    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c2de21_e6a2_4075_838d_562a35d48252.slice/crio-38c9dd8762e0ad7f3adc79ce3e279c778dbc81df002ba4f5dd671fd17e67815f WatchSource:0}: Error finding container 38c9dd8762e0ad7f3adc79ce3e279c778dbc81df002ba4f5dd671fd17e67815f: Status 404 returned error can't find the container with id 38c9dd8762e0ad7f3adc79ce3e279c778dbc81df002ba4f5dd671fd17e67815f
Oct 30 01:31:24.579853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.579828    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c066f1b-c608-42af-800c-7167e4c48a80" path="/var/lib/kubelet/pods/2c066f1b-c608-42af-800c-7167e4c48a80/volumes"
Oct 30 01:31:24.580217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:24.580201    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"]
Oct 30 01:31:25.253525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:25.253446    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs" event={"ID":"50c2de21-e6a2-4075-838d-562a35d48252","Type":"ContainerStarted","Data":"0a4926bad7aa7bff861203637d0fdec7077fab4f4aedd90fce138808f2269c26"}
Oct 30 01:31:25.253748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:25.253550    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs" event={"ID":"50c2de21-e6a2-4075-838d-562a35d48252","Type":"ContainerStarted","Data":"38c9dd8762e0ad7f3adc79ce3e279c778dbc81df002ba4f5dd671fd17e67815f"}
Oct 30 01:31:25.301949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:25.301879    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs" podStartSLOduration=2.301860746 podStartE2EDuration="2.301860746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:25.298929638 +0000 UTC m=+2926.846620134" watchObservedRunningTime="2026-10-30 01:31:25.301860746 +0000 UTC m=+2926.849551243"
Oct 30 01:31:25.307956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:25.307920    5104 patch_prober.go:29] interesting pod/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:31:25.308112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:25.307983    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:31:26.262191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:26.262157    5104 generic.go:356] "Generic (PLEG): container finished" podID="50c2de21-e6a2-4075-838d-562a35d48252" containerID="0a4926bad7aa7bff861203637d0fdec7077fab4f4aedd90fce138808f2269c26" exitCode=0
Oct 30 01:31:26.262384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:26.262238    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs" event={"ID":"50c2de21-e6a2-4075-838d-562a35d48252","Type":"ContainerDied","Data":"0a4926bad7aa7bff861203637d0fdec7077fab4f4aedd90fce138808f2269c26"}
Oct 30 01:31:27.439639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.439610    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:27.485969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.485939    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-openshift-ca\") pod \"50c2de21-e6a2-4075-838d-562a35d48252\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") "
Oct 30 01:31:27.486148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.486040    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c2de21-e6a2-4075-838d-562a35d48252-kube-api-access-fn58b\" (UniqueName: \"kubernetes.io/projected/50c2de21-e6a2-4075-838d-562a35d48252-kube-api-access-fn58b\") pod \"50c2de21-e6a2-4075-838d-562a35d48252\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") "
Oct 30 01:31:27.486148 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.486137    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-xks-ca\") pod \"50c2de21-e6a2-4075-838d-562a35d48252\" (UID: \"50c2de21-e6a2-4075-838d-562a35d48252\") "
Oct 30 01:31:27.486345 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.486318    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-openshift-ca" pod "50c2de21-e6a2-4075-838d-562a35d48252" (UID: "50c2de21-e6a2-4075-838d-562a35d48252"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:27.486596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.486571    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:27.488202 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.488171    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-xks-ca" pod "50c2de21-e6a2-4075-838d-562a35d48252" (UID: "50c2de21-e6a2-4075-838d-562a35d48252"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:27.488484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.488437    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c2de21-e6a2-4075-838d-562a35d48252-kube-api-access-fn58b" pod "50c2de21-e6a2-4075-838d-562a35d48252" (UID: "50c2de21-e6a2-4075-838d-562a35d48252"). InnerVolumeSpecName "kube-api-access-fn58b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:27.587824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.587742    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c2de21-e6a2-4075-838d-562a35d48252-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:27.587824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:27.587774    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn58b\" (UniqueName: \"kubernetes.io/projected/50c2de21-e6a2-4075-838d-562a35d48252-kube-api-access-fn58b\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.280716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.280680    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"
Oct 30 01:31:28.280917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.280719    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs" event={"ID":"50c2de21-e6a2-4075-838d-562a35d48252","Type":"ContainerDied","Data":"38c9dd8762e0ad7f3adc79ce3e279c778dbc81df002ba4f5dd671fd17e67815f"}
Oct 30 01:31:28.280917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.280783    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38c9dd8762e0ad7f3adc79ce3e279c778dbc81df002ba4f5dd671fd17e67815f"
Oct 30 01:31:28.749338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.749312    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:31:28.804668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804627    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-envoy\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.804849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804687    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-certs\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.804849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804728    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-podinfo\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.804849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804776    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-credential-socket\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.804849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804814    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-data\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.805076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804887    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-token\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.805076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804938    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-kube-api-access-nbscn\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-kube-api-access-nbscn\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.805076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.804990    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-socket\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.805076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805025    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-certs" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:28.805076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805048    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istiod-ca-cert\") pod \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\" (UID: \"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3\") "
Oct 30 01:31:28.805332 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805181    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-credential-socket" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:28.805579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805555    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.805705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805581    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.805705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805671    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-data" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:28.805815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805713    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istiod-ca-cert" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:28.805815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.805796    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-socket" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:28.807686 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.807658    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-envoy" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:28.807865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.807837    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-kube-api-access-nbscn" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "kube-api-access-nbscn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:28.807993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.807967    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-token" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:28.809604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.809575    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-podinfo" pod "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" (UID: "5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:31:28.907418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.907324    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.907418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.907371    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.907418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.907386    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.907418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.907400    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.907418 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.907413    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.907794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.907428    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.907794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.907442    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbscn\" (UniqueName: \"kubernetes.io/projected/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3-kube-api-access-nbscn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:28.912670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.912633    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"]
Oct 30 01:31:28.933061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:28.933021    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"]
Oct 30 01:31:29.049832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.049797    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:31:29.291614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.291571    5104 generic.go:356] "Generic (PLEG): container finished" podID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerID="3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c" exitCode=0
Oct 30 01:31:29.291790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.291626    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"
Oct 30 01:31:29.291790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.291637    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" event={"ID":"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3","Type":"ContainerDied","Data":"3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c"}
Oct 30 01:31:29.291790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.291697    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx" event={"ID":"5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3","Type":"ContainerDied","Data":"47e1b72781929e3fa6f2d6f6175362bf892a74a45a588fa8dba34c8ea7a8a652"}
Oct 30 01:31:29.291790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.291734    5104 scope.go:122] "RemoveContainer" containerID="3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c"
Oct 30 01:31:29.304558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.304529    5104 scope.go:122] "RemoveContainer" containerID="6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511"
Oct 30 01:31:29.315375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.315351    5104 scope.go:122] "RemoveContainer" containerID="3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c"
Oct 30 01:31:29.315716 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:29.315696    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c\": container with ID starting with 3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c not found: ID does not exist" containerID="3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c"
Oct 30 01:31:29.315826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.315733    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c"} err="failed to get container status \"3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c\": rpc error: code = NotFound desc = could not find container \"3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c\": container with ID starting with 3810af365581760e789a1dcd5b0e454dcb557f331d3091deb2035f63b44cdf3c not found: ID does not exist"
Oct 30 01:31:29.315826 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.315755    5104 scope.go:122] "RemoveContainer" containerID="6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511"
Oct 30 01:31:29.316032 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:29.316004    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511\": container with ID starting with 6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511 not found: ID does not exist" containerID="6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511"
Oct 30 01:31:29.316087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.316043    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511"} err="failed to get container status \"6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511\": rpc error: code = NotFound desc = could not find container \"6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511\": container with ID starting with 6c70b02bdaf6c18b45dfd967d1b69e2b412cf72af8fff3fca681361ed31e3511 not found: ID does not exist"
Oct 30 01:31:29.347350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.347313    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"]
Oct 30 01:31:29.352719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:29.352676    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-679bd2-openshift-default-fb6c76fc-bncwx"]
Oct 30 01:31:30.577154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:30.577116    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" path="/var/lib/kubelet/pods/5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3/volumes"
Oct 30 01:31:33.913720 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:33.913672    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" podUID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" containerName="payload-processing" containerID="cri-o://098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860" gracePeriod=25
Oct 30 01:31:33.945160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:33.933427    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" podUID="d95743e3-96fd-481b-99e6-88257ec2d360" containerName="payload-pre-processing" containerID="cri-o://eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78" gracePeriod=25
Oct 30 01:31:34.494957 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:34.494914    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"]
Oct 30 01:31:34.495193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:34.495005    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"]
Oct 30 01:31:34.496260 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:34.496224    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-886adb-cbhhg"]
Oct 30 01:31:34.505587 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:34.505561    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"]
Oct 30 01:31:34.580995 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:34.580950    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83a401d-3694-44e7-8ad5-04bd3940b107" path="/var/lib/kubelet/pods/f83a401d-3694-44e7-8ad5-04bd3940b107/volumes"
Oct 30 01:31:36.749436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:36.749403    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"]
Oct 30 01:31:36.759834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:36.759798    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d81f0a-wjhgs"]
Oct 30 01:31:36.777998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:36.777955    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" podUID="d95743e3-96fd-481b-99e6-88257ec2d360" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:36.778164 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:36.778015    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" podUID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:38.578132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:38.578095    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c2de21-e6a2-4075-838d-562a35d48252" path="/var/lib/kubelet/pods/50c2de21-e6a2-4075-838d-562a35d48252/volumes"
Oct 30 01:31:39.105604 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:39.105553    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:31:39.527087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:39.526627    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" podUID="4352416c-a1ef-417c-9052-6274a29fd052" containerName="payload-pre-processing" containerID="cri-o://43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d" gracePeriod=25
Oct 30 01:31:39.528243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:39.527814    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" podUID="c24262f6-4765-4404-86c0-87cbcd13dd97" containerName="payload-processing" containerID="cri-o://768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae" gracePeriod=25
Oct 30 01:31:39.530196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:39.530171    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:31:42.865959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:42.865920    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" podUID="4352416c-a1ef-417c-9052-6274a29fd052" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:42.865959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:42.865933    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" podUID="c24262f6-4765-4404-86c0-87cbcd13dd97" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:44.159769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.159733    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"]
Oct 30 01:31:44.160174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.159947    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy" containerID="cri-o://1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5" gracePeriod=30
Oct 30 01:31:44.790221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.790175    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"]
Oct 30 01:31:44.790860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.790840    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy"
Oct 30 01:31:44.790922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.790863    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy"
Oct 30 01:31:44.790922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.790881    5104 cpu_manager.go:414] "containerMap: removing container" podUID="50c2de21-e6a2-4075-838d-562a35d48252" containerName="revoke-keys"
Oct 30 01:31:44.790922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.790886    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c2de21-e6a2-4075-838d-562a35d48252" containerName="revoke-keys"
Oct 30 01:31:44.791036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.790994    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy"
Oct 30 01:31:44.791036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.791003    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c2de21-e6a2-4075-838d-562a35d48252" containerName="revoke-keys"
Oct 30 01:31:44.791036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.791014    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy"
Oct 30 01:31:44.795819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.795796    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.827351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.827301    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"]
Oct 30 01:31:44.833889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.833854    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:31:44.833889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.833879    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:31:44.833889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.833879    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:31:44.834144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.833854    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:44.882304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.882271    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.882511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.882316    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.882511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.882410    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmlt\" (UniqueName: \"kubernetes.io/projected/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-kube-api-access-2pmlt\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.983189 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.983145    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmlt\" (UniqueName: \"kubernetes.io/projected/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-kube-api-access-2pmlt\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.983412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.983282    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.983412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.983330    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.984135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.984109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:44.985908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:44.985882    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:45.006809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:45.006779    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmlt\" (UniqueName: \"kubernetes.io/projected/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-kube-api-access-2pmlt\") pod \"maas-api-revoke-keys-e2e-worker-w3-82b647-8p279\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:45.111340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:45.111238    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:45.416603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:45.416576    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"]
Oct 30 01:31:45.419325 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:31:45.419296    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3bf0c6_928f_4886_bcde_3805f4ee82b8.slice/crio-ed4d4cb9c8474485dfbdffde19f8c317f0afd30fff942efb051f1f841cb6ca43 WatchSource:0}: Error finding container ed4d4cb9c8474485dfbdffde19f8c317f0afd30fff942efb051f1f841cb6ca43: Status 404 returned error can't find the container with id ed4d4cb9c8474485dfbdffde19f8c317f0afd30fff942efb051f1f841cb6ca43
Oct 30 01:31:45.453674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:45.453633    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279" event={"ID":"cc3bf0c6-928f-4886-bcde-3805f4ee82b8","Type":"ContainerStarted","Data":"ed4d4cb9c8474485dfbdffde19f8c317f0afd30fff942efb051f1f841cb6ca43"}
Oct 30 01:31:46.339336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.339299    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"]
Oct 30 01:31:46.339661 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.339582    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy" containerID="cri-o://79d66a9f60797ea53ea5f82191493a404fae9ce49790edd55aab8fa9251aab79" gracePeriod=30
Oct 30 01:31:46.476777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.472910    5104 generic.go:356] "Generic (PLEG): container finished" podID="cc3bf0c6-928f-4886-bcde-3805f4ee82b8" containerID="91e10102550c9972d83552d9ce2bd2c4bbcb02a6b82a983231671c4b22bdc129" exitCode=0
Oct 30 01:31:46.476777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.473145    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279" event={"ID":"cc3bf0c6-928f-4886-bcde-3805f4ee82b8","Type":"ContainerDied","Data":"91e10102550c9972d83552d9ce2bd2c4bbcb02a6b82a983231671c4b22bdc129"}
Oct 30 01:31:46.778208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.778119    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" podUID="d95743e3-96fd-481b-99e6-88257ec2d360" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:46.778366 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.778133    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" podUID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:46.835668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.835629    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"]
Oct 30 01:31:46.836565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.836535    5104 cpu_manager.go:414] "containerMap: removing container" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy"
Oct 30 01:31:46.836565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.836560    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf5c036-e9ce-48ed-95b8-b06cc3c1e4f3" containerName="istio-proxy"
Oct 30 01:31:46.840856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.840830    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:46.857074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.857042    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"]
Oct 30 01:31:46.907066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.907025    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5c2\" (UniqueName: \"kubernetes.io/projected/161f57e6-e06e-4fd3-8254-a99af6019ce0-kube-api-access-wh5c2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:46.907304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.907085    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:46.907304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:46.907267    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.008640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.008602    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.008858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.008724    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5c2\" (UniqueName: \"kubernetes.io/projected/161f57e6-e06e-4fd3-8254-a99af6019ce0-kube-api-access-wh5c2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.008858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.008765    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.009575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.009544    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.011171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.011149    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.030691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.030631    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5c2\" (UniqueName: \"kubernetes.io/projected/161f57e6-e06e-4fd3-8254-a99af6019ce0-kube-api-access-wh5c2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.160393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.160360    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:47.472196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.472153    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"]
Oct 30 01:31:47.472609 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:31:47.472581    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161f57e6_e06e_4fd3_8254_a99af6019ce0.slice/crio-63f0dc8d8cd80fd0ce1ea1a28d7133fc0b3cb8f299c7b4679e75e163b63916d3 WatchSource:0}: Error finding container 63f0dc8d8cd80fd0ce1ea1a28d7133fc0b3cb8f299c7b4679e75e163b63916d3: Status 404 returned error can't find the container with id 63f0dc8d8cd80fd0ce1ea1a28d7133fc0b3cb8f299c7b4679e75e163b63916d3
Oct 30 01:31:47.491346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.491308    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx" event={"ID":"161f57e6-e06e-4fd3-8254-a99af6019ce0","Type":"ContainerStarted","Data":"63f0dc8d8cd80fd0ce1ea1a28d7133fc0b3cb8f299c7b4679e75e163b63916d3"}
Oct 30 01:31:47.685231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.685190    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:47.717543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.717512    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-kube-api-access-2pmlt\" (UniqueName: \"kubernetes.io/projected/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-kube-api-access-2pmlt\") pod \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") "
Oct 30 01:31:47.717751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.717568    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-xks-ca\") pod \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") "
Oct 30 01:31:47.717751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.717668    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-openshift-ca\") pod \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\" (UID: \"cc3bf0c6-928f-4886-bcde-3805f4ee82b8\") "
Oct 30 01:31:47.718007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.717985    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-openshift-ca" pod "cc3bf0c6-928f-4886-bcde-3805f4ee82b8" (UID: "cc3bf0c6-928f-4886-bcde-3805f4ee82b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:47.718187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.718168    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:47.719915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.719851    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-xks-ca" pod "cc3bf0c6-928f-4886-bcde-3805f4ee82b8" (UID: "cc3bf0c6-928f-4886-bcde-3805f4ee82b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:47.720045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.719967    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-kube-api-access-2pmlt" pod "cc3bf0c6-928f-4886-bcde-3805f4ee82b8" (UID: "cc3bf0c6-928f-4886-bcde-3805f4ee82b8"). InnerVolumeSpecName "kube-api-access-2pmlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:47.819748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.819697    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pmlt\" (UniqueName: \"kubernetes.io/projected/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-kube-api-access-2pmlt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:47.819748 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:47.819744    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc3bf0c6-928f-4886-bcde-3805f4ee82b8-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:48.501544 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:48.501504    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx" event={"ID":"161f57e6-e06e-4fd3-8254-a99af6019ce0","Type":"ContainerStarted","Data":"16ca2e165bda7e9ec6349ba1d9bcb977f9d7c5d41a15096bb048791351e1de78"}
Oct 30 01:31:48.503183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:48.503161    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"
Oct 30 01:31:48.503337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:48.503161    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279" event={"ID":"cc3bf0c6-928f-4886-bcde-3805f4ee82b8","Type":"ContainerDied","Data":"ed4d4cb9c8474485dfbdffde19f8c317f0afd30fff942efb051f1f841cb6ca43"}
Oct 30 01:31:48.503337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:48.503289    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed4d4cb9c8474485dfbdffde19f8c317f0afd30fff942efb051f1f841cb6ca43"
Oct 30 01:31:48.542705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:48.542451    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx" podStartSLOduration=2.542432346 podStartE2EDuration="2.542432346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:48.540348671 +0000 UTC m=+2950.088039166" watchObservedRunningTime="2026-10-30 01:31:48.542432346 +0000 UTC m=+2950.090122840"
Oct 30 01:31:49.458899 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.458872    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:31:49.516897 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.516853    5104 generic.go:356] "Generic (PLEG): container finished" podID="a316d84a-4399-420a-82cf-74c36d1a431f" containerID="1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5" exitCode=0
Oct 30 01:31:49.517368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.517210    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" event={"ID":"a316d84a-4399-420a-82cf-74c36d1a431f","Type":"ContainerDied","Data":"1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5"}
Oct 30 01:31:49.517368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.517279    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk" event={"ID":"a316d84a-4399-420a-82cf-74c36d1a431f","Type":"ContainerDied","Data":"d7ce94e330d39a2a516f648266aa9fd0158d50368c2fd026067a19dcf80f6728"}
Oct 30 01:31:49.517368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.517316    5104 scope.go:122] "RemoveContainer" containerID="1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5"
Oct 30 01:31:49.517749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.517047    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"
Oct 30 01:31:49.521590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.521296    5104 generic.go:356] "Generic (PLEG): container finished" podID="161f57e6-e06e-4fd3-8254-a99af6019ce0" containerID="16ca2e165bda7e9ec6349ba1d9bcb977f9d7c5d41a15096bb048791351e1de78" exitCode=0
Oct 30 01:31:49.521590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.521398    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx" event={"ID":"161f57e6-e06e-4fd3-8254-a99af6019ce0","Type":"ContainerDied","Data":"16ca2e165bda7e9ec6349ba1d9bcb977f9d7c5d41a15096bb048791351e1de78"}
Oct 30 01:31:49.534174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.534126    5104 scope.go:122] "RemoveContainer" containerID="1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5"
Oct 30 01:31:49.541640 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:49.541577    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5\": container with ID starting with 1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5 not found: ID does not exist" containerID="1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5"
Oct 30 01:31:49.542001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.541955    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5"} err="failed to get container status \"1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5\": rpc error: code = NotFound desc = could not find container \"1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5\": container with ID starting with 1255178e4fa887eae085888b74f8e742c5450bd78363c2e8250d1c2d19dfd6f5 not found: ID does not exist"
Oct 30 01:31:49.542693 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.542663    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a316d84a-4399-420a-82cf-74c36d1a431f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316d84a-4399-420a-82cf-74c36d1a431f-istio-podinfo\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.542821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.542735    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-credential-socket\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.542821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.542782    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-envoy\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.542961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.542824    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-kube-api-access-59tnt\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-kube-api-access-59tnt\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.542961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.542872    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-data\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.542961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.542931    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a316d84a-4399-420a-82cf-74c36d1a431f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316d84a-4399-420a-82cf-74c36d1a431f-istiod-ca-cert\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.543144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.542993    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-certs\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.543144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.543073    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-istio-token\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-istio-token\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.543144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.543135    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-socket\") pod \"a316d84a-4399-420a-82cf-74c36d1a431f\" (UID: \"a316d84a-4399-420a-82cf-74c36d1a431f\") "
Oct 30 01:31:49.546309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.546277    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-credential-socket" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:49.546432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.546274    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-data" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:49.546432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.546381    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-certs" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:49.546606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.546535    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-socket" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:49.546784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.546758    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a316d84a-4399-420a-82cf-74c36d1a431f-istiod-ca-cert" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:49.549122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.549084    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-kube-api-access-59tnt" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "kube-api-access-59tnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:49.549885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.549857    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a316d84a-4399-420a-82cf-74c36d1a431f-istio-podinfo" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:31:49.549885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.549874    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-envoy" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:49.551576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.551554    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-istio-token" pod "a316d84a-4399-420a-82cf-74c36d1a431f" (UID: "a316d84a-4399-420a-82cf-74c36d1a431f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:49.645072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645039    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645070    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645080    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316d84a-4399-420a-82cf-74c36d1a431f-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645089    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645098    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645107    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59tnt\" (UniqueName: \"kubernetes.io/projected/a316d84a-4399-420a-82cf-74c36d1a431f-kube-api-access-59tnt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645116    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645125    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316d84a-4399-420a-82cf-74c36d1a431f-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.645326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.645133    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316d84a-4399-420a-82cf-74c36d1a431f-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:49.884445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.884406    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"]
Oct 30 01:31:49.890134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:49.890104    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-q62nk"]
Oct 30 01:31:50.577437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.577403    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" path="/var/lib/kubelet/pods/a316d84a-4399-420a-82cf-74c36d1a431f/volumes"
Oct 30 01:31:50.700064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.700031    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:50.757863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.757822    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-openshift-ca\") pod \"161f57e6-e06e-4fd3-8254-a99af6019ce0\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") "
Oct 30 01:31:50.758045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.757967    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-xks-ca\") pod \"161f57e6-e06e-4fd3-8254-a99af6019ce0\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") "
Oct 30 01:31:50.758045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.758034    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161f57e6-e06e-4fd3-8254-a99af6019ce0-kube-api-access-wh5c2\" (UniqueName: \"kubernetes.io/projected/161f57e6-e06e-4fd3-8254-a99af6019ce0-kube-api-access-wh5c2\") pod \"161f57e6-e06e-4fd3-8254-a99af6019ce0\" (UID: \"161f57e6-e06e-4fd3-8254-a99af6019ce0\") "
Oct 30 01:31:50.758298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.758264    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-openshift-ca" pod "161f57e6-e06e-4fd3-8254-a99af6019ce0" (UID: "161f57e6-e06e-4fd3-8254-a99af6019ce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:50.758497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.758443    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:50.760142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.760113    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-xks-ca" pod "161f57e6-e06e-4fd3-8254-a99af6019ce0" (UID: "161f57e6-e06e-4fd3-8254-a99af6019ce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:50.760739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.760716    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161f57e6-e06e-4fd3-8254-a99af6019ce0-kube-api-access-wh5c2" pod "161f57e6-e06e-4fd3-8254-a99af6019ce0" (UID: "161f57e6-e06e-4fd3-8254-a99af6019ce0"). InnerVolumeSpecName "kube-api-access-wh5c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:50.859850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.859747    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f57e6-e06e-4fd3-8254-a99af6019ce0-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:50.859850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:50.859783    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5c2\" (UniqueName: \"kubernetes.io/projected/161f57e6-e06e-4fd3-8254-a99af6019ce0-kube-api-access-wh5c2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.557445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.557403    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx" event={"ID":"161f57e6-e06e-4fd3-8254-a99af6019ce0","Type":"ContainerDied","Data":"63f0dc8d8cd80fd0ce1ea1a28d7133fc0b3cb8f299c7b4679e75e163b63916d3"}
Oct 30 01:31:51.557638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.557484    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63f0dc8d8cd80fd0ce1ea1a28d7133fc0b3cb8f299c7b4679e75e163b63916d3"
Oct 30 01:31:51.557638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.557552    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"
Oct 30 01:31:51.564428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.564391    5104 generic.go:356] "Generic (PLEG): container finished" podID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerID="79d66a9f60797ea53ea5f82191493a404fae9ce49790edd55aab8fa9251aab79" exitCode=0
Oct 30 01:31:51.564671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.564511    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" event={"ID":"1d563dff-75e3-48e2-b6ca-4dd41a6cca56","Type":"ContainerDied","Data":"79d66a9f60797ea53ea5f82191493a404fae9ce49790edd55aab8fa9251aab79"}
Oct 30 01:31:51.564671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.564584    5104 scope.go:122] "RemoveContainer" containerID="f3631caf395c09e7e20feaf56189b9c3db723ec3bfe09a8007ebcc93e2ce279f"
Oct 30 01:31:51.631073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.631048    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:31:51.668355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668323    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-kube-api-access-xph48\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-kube-api-access-xph48\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668416    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-socket\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668547    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istiod-ca-cert\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668619    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-podinfo\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668657    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-credential-socket\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668692    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-token\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-token\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668742    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-certs\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668751    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-socket" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:51.668875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668777    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-data\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.668875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668809    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-envoy\") pod \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\" (UID: \"1d563dff-75e3-48e2-b6ca-4dd41a6cca56\") "
Oct 30 01:31:51.669060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668902    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istiod-ca-cert" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:51.669115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668983    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-credential-socket" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:51.669115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.668994    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-certs" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:51.669230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.669142    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-data" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:51.669315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.669295    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.669393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.669317    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.669393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.669334    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.669393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.669348    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.669393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.669359    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.671117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.671087    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-podinfo" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:31:51.671309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.671117    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-token" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:51.671309 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.671164    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-kube-api-access-xph48" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "kube-api-access-xph48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:51.671698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.671671    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-envoy" pod "1d563dff-75e3-48e2-b6ca-4dd41a6cca56" (UID: "1d563dff-75e3-48e2-b6ca-4dd41a6cca56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:51.770681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.770640    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.770681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.770674    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.770681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.770684    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:51.770938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:51.770692    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xph48\" (UniqueName: \"kubernetes.io/projected/1d563dff-75e3-48e2-b6ca-4dd41a6cca56-kube-api-access-xph48\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:52.573812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:52.573783    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"
Oct 30 01:31:52.576780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:52.576739    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66" event={"ID":"1d563dff-75e3-48e2-b6ca-4dd41a6cca56","Type":"ContainerDied","Data":"68f97f90abbbe74aee8dc4737d04d5748974474d35a9537e87645071c3dc0ef0"}
Oct 30 01:31:52.576953 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:52.576802    5104 scope.go:122] "RemoveContainer" containerID="79d66a9f60797ea53ea5f82191493a404fae9ce49790edd55aab8fa9251aab79"
Oct 30 01:31:52.628135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:52.628105    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"]
Oct 30 01:31:52.638730 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:52.638699    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-94l66"]
Oct 30 01:31:52.865854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:52.865761    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" podUID="c24262f6-4765-4404-86c0-87cbcd13dd97" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:52.865854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:52.865817    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" podUID="4352416c-a1ef-417c-9052-6274a29fd052" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:54.209133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.209101    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:31:54.254663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.254634    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:31:54.298880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.298847    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-kube-api-access-dt9qm\" (UniqueName: \"kubernetes.io/projected/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-kube-api-access-dt9qm\") pod \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") "
Oct 30 01:31:54.299056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.298892    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95743e3-96fd-481b-99e6-88257ec2d360-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95743e3-96fd-481b-99e6-88257ec2d360-plugins-config-volume\") pod \"d95743e3-96fd-481b-99e6-88257ec2d360\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") "
Oct 30 01:31:54.299056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.298925    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95743e3-96fd-481b-99e6-88257ec2d360-kube-api-access-lqvk6\" (UniqueName: \"kubernetes.io/projected/d95743e3-96fd-481b-99e6-88257ec2d360-kube-api-access-lqvk6\") pod \"d95743e3-96fd-481b-99e6-88257ec2d360\" (UID: \"d95743e3-96fd-481b-99e6-88257ec2d360\") "
Oct 30 01:31:54.299056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.298974    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-plugins-config-volume\") pod \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\" (UID: \"4fc11c49-e9b1-4649-a0cf-2917bbff63b7\") "
Oct 30 01:31:54.299396 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.299369    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95743e3-96fd-481b-99e6-88257ec2d360-plugins-config-volume" pod "d95743e3-96fd-481b-99e6-88257ec2d360" (UID: "d95743e3-96fd-481b-99e6-88257ec2d360"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:54.299545 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.299424    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-plugins-config-volume" pod "4fc11c49-e9b1-4649-a0cf-2917bbff63b7" (UID: "4fc11c49-e9b1-4649-a0cf-2917bbff63b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:54.302439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.302408    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-kube-api-access-dt9qm" pod "4fc11c49-e9b1-4649-a0cf-2917bbff63b7" (UID: "4fc11c49-e9b1-4649-a0cf-2917bbff63b7"). InnerVolumeSpecName "kube-api-access-dt9qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:54.302718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.302691    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95743e3-96fd-481b-99e6-88257ec2d360-kube-api-access-lqvk6" pod "d95743e3-96fd-481b-99e6-88257ec2d360" (UID: "d95743e3-96fd-481b-99e6-88257ec2d360"). InnerVolumeSpecName "kube-api-access-lqvk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:54.401139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.401001    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt9qm\" (UniqueName: \"kubernetes.io/projected/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-kube-api-access-dt9qm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:54.401139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.401036    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95743e3-96fd-481b-99e6-88257ec2d360-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:54.401139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.401051    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqvk6\" (UniqueName: \"kubernetes.io/projected/d95743e3-96fd-481b-99e6-88257ec2d360-kube-api-access-lqvk6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:54.401139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.401060    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc11c49-e9b1-4649-a0cf-2917bbff63b7-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:54.577606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.577527    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" path="/var/lib/kubelet/pods/1d563dff-75e3-48e2-b6ca-4dd41a6cca56/volumes"
Oct 30 01:31:54.601438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.601400    5104 generic.go:356] "Generic (PLEG): container finished" podID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" containerID="098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860" exitCode=0
Oct 30 01:31:54.601660 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.601636    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"
Oct 30 01:31:54.602144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.601718    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" event={"ID":"4fc11c49-e9b1-4649-a0cf-2917bbff63b7","Type":"ContainerDied","Data":"098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860"}
Oct 30 01:31:54.602302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.602218    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m" event={"ID":"4fc11c49-e9b1-4649-a0cf-2917bbff63b7","Type":"ContainerDied","Data":"61435ee4f952ee2d1aa723ece5157cd7a20c2fad4258be3287eca08d5c7837ae"}
Oct 30 01:31:54.602302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.602255    5104 scope.go:122] "RemoveContainer" containerID="098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860"
Oct 30 01:31:54.603610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.603509    5104 generic.go:356] "Generic (PLEG): container finished" podID="d95743e3-96fd-481b-99e6-88257ec2d360" containerID="eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78" exitCode=0
Oct 30 01:31:54.603610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.603563    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" event={"ID":"d95743e3-96fd-481b-99e6-88257ec2d360","Type":"ContainerDied","Data":"eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78"}
Oct 30 01:31:54.603610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.603596    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"
Oct 30 01:31:54.603805 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.603602    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48" event={"ID":"d95743e3-96fd-481b-99e6-88257ec2d360","Type":"ContainerDied","Data":"09a844023c173a40c2178be5f553d1f313150171eb37e6b17e56e1c21eeec9cc"}
Oct 30 01:31:54.614032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.614000    5104 scope.go:122] "RemoveContainer" containerID="098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860"
Oct 30 01:31:54.614501 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:54.614451    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860\": container with ID starting with 098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860 not found: ID does not exist" containerID="098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860"
Oct 30 01:31:54.614612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.614528    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860"} err="failed to get container status \"098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860\": rpc error: code = NotFound desc = could not find container \"098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860\": container with ID starting with 098e8302482f3b362edf0f200fe569f7d2760fd7ef04c07841074b92117e0860 not found: ID does not exist"
Oct 30 01:31:54.614612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.614576    5104 scope.go:122] "RemoveContainer" containerID="eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78"
Oct 30 01:31:54.628657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.625582    5104 scope.go:122] "RemoveContainer" containerID="eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78"
Oct 30 01:31:54.629033 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:31:54.629004    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78\": container with ID starting with eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78 not found: ID does not exist" containerID="eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78"
Oct 30 01:31:54.629105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.629042    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78"} err="failed to get container status \"eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78\": rpc error: code = NotFound desc = could not find container \"eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78\": container with ID starting with eded9fa4042d3da6bb0f333b870d37adcd893e7f5273a8c073fe4af908648f78 not found: ID does not exist"
Oct 30 01:31:54.656997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.656958    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"]
Oct 30 01:31:54.660067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.660038    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-886adb-5bdf49f978-nft8m"]
Oct 30 01:31:54.694866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.694831    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"]
Oct 30 01:31:54.700217 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.700184    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-886adb-74d5cf5c6c-cql48"]
Oct 30 01:31:54.872691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.872598    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"]
Oct 30 01:31:54.906491 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:54.906438    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"]
Oct 30 01:31:55.036066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:55.036036    5104 scope.go:122] "RemoveContainer" containerID="7f52eb0ac3f4a5b27799f5436c7f6c3b7c3796055aa7249b1e2788355e9e726f"
Oct 30 01:31:55.046743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:55.046722    5104 scope.go:122] "RemoveContainer" containerID="05966e0160a5291b5f6ee06e181b82f1d950fff6b37389496ed0911652b967e0"
Oct 30 01:31:55.056666 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:55.056509    5104 scope.go:122] "RemoveContainer" containerID="fcc571f8f8f2fbde62261c7574a62d05ccff07344bc3349087cfe7aaeeeda1a9"
Oct 30 01:31:55.066269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:55.066250    5104 scope.go:122] "RemoveContainer" containerID="b3d330eb91abce89ffbcae1ce08bdc319c3b7d4af4f23ae0155833e57825771b"
Oct 30 01:31:56.577540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:56.577501    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" path="/var/lib/kubelet/pods/4fc11c49-e9b1-4649-a0cf-2917bbff63b7/volumes"
Oct 30 01:31:56.577976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:56.577936    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95743e3-96fd-481b-99e6-88257ec2d360" path="/var/lib/kubelet/pods/d95743e3-96fd-481b-99e6-88257ec2d360/volumes"
Oct 30 01:31:59.790553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.790385    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"]
Oct 30 01:31:59.791955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.791924    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy"
Oct 30 01:31:59.792115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792098    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy"
Oct 30 01:31:59.792256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792244    5104 cpu_manager.go:414] "containerMap: removing container" podUID="cc3bf0c6-928f-4886-bcde-3805f4ee82b8" containerName="revoke-keys"
Oct 30 01:31:59.792357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792345    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3bf0c6-928f-4886-bcde-3805f4ee82b8" containerName="revoke-keys"
Oct 30 01:31:59.792468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792441    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" containerName="payload-processing"
Oct 30 01:31:59.792586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792572    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" containerName="payload-processing"
Oct 30 01:31:59.792692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792681    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy"
Oct 30 01:31:59.792779 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792768    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy"
Oct 30 01:31:59.792873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792861    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d95743e3-96fd-481b-99e6-88257ec2d360" containerName="payload-pre-processing"
Oct 30 01:31:59.792955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.792945    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95743e3-96fd-481b-99e6-88257ec2d360" containerName="payload-pre-processing"
Oct 30 01:31:59.793055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793045    5104 cpu_manager.go:414] "containerMap: removing container" podUID="161f57e6-e06e-4fd3-8254-a99af6019ce0" containerName="revoke-keys"
Oct 30 01:31:59.793144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793133    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161f57e6-e06e-4fd3-8254-a99af6019ce0" containerName="revoke-keys"
Oct 30 01:31:59.793372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793356    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3bf0c6-928f-4886-bcde-3805f4ee82b8" containerName="revoke-keys"
Oct 30 01:31:59.793505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793492    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc11c49-e9b1-4649-a0cf-2917bbff63b7" containerName="payload-processing"
Oct 30 01:31:59.793611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793601    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a316d84a-4399-420a-82cf-74c36d1a431f" containerName="istio-proxy"
Oct 30 01:31:59.793709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793698    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95743e3-96fd-481b-99e6-88257ec2d360" containerName="payload-pre-processing"
Oct 30 01:31:59.793792 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793782    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="161f57e6-e06e-4fd3-8254-a99af6019ce0" containerName="revoke-keys"
Oct 30 01:31:59.793882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793871    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy"
Oct 30 01:31:59.793971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.793960    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy"
Oct 30 01:31:59.799540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.798154    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.821508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.816806    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"]
Oct 30 01:31:59.834487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.832915    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:31:59.834487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.832970    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:59.834487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.833051    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:31:59.834487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.833153    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:31:59.834835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.834662    5104 cpu_manager.go:414] "containerMap: removing container" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy"
Oct 30 01:31:59.834835 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.834685    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d563dff-75e3-48e2-b6ca-4dd41a6cca56" containerName="istio-proxy"
Oct 30 01:31:59.850961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.850915    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"]
Oct 30 01:31:59.851259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.851225    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"]
Oct 30 01:31:59.851595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.851575    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:31:59.867212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.867172    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.867388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.867362    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.867449 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.867406    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/6db7542b-893d-4d17-85e2-ccadaf949201-kube-api-access-7czs8\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.873210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.873180    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" podUID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" containerName="payload-processing" containerID="cri-o://d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311" gracePeriod=25
Oct 30 01:31:59.908919 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.907720    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" podUID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" containerName="payload-pre-processing" containerID="cri-o://3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996" gracePeriod=25
Oct 30 01:31:59.925158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.925123    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:31:59.965925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.965895    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:31:59.967983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.967953    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24262f6-4765-4404-86c0-87cbcd13dd97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24262f6-4765-4404-86c0-87cbcd13dd97-plugins-config-volume\") pod \"c24262f6-4765-4404-86c0-87cbcd13dd97\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") "
Oct 30 01:31:59.968146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968054    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24262f6-4765-4404-86c0-87cbcd13dd97-kube-api-access-tb6d6\" (UniqueName: \"kubernetes.io/projected/c24262f6-4765-4404-86c0-87cbcd13dd97-kube-api-access-tb6d6\") pod \"c24262f6-4765-4404-86c0-87cbcd13dd97\" (UID: \"c24262f6-4765-4404-86c0-87cbcd13dd97\") "
Oct 30 01:31:59.968371 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968346    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24262f6-4765-4404-86c0-87cbcd13dd97-plugins-config-volume" pod "c24262f6-4765-4404-86c0-87cbcd13dd97" (UID: "c24262f6-4765-4404-86c0-87cbcd13dd97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:59.968507 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.968560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968512    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:31:59.968560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968549    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.968685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968577    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/6db7542b-893d-4d17-85e2-ccadaf949201-kube-api-access-7czs8\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.968685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968601    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4fh2\" (UniqueName: \"kubernetes.io/projected/abdc9c46-8122-45a4-9080-cba7ab8bd44e-kube-api-access-k4fh2\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:31:59.968795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968708    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:31:59.968856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.968802    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24262f6-4765-4404-86c0-87cbcd13dd97-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:31:59.969795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.969762    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.970946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.970917    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24262f6-4765-4404-86c0-87cbcd13dd97-kube-api-access-tb6d6" pod "c24262f6-4765-4404-86c0-87cbcd13dd97" (UID: "c24262f6-4765-4404-86c0-87cbcd13dd97"). InnerVolumeSpecName "kube-api-access-tb6d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:59.971289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.971263    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:31:59.987397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:31:59.987366    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/6db7542b-893d-4d17-85e2-ccadaf949201-kube-api-access-7czs8\") pod \"maas-api-revoke-sub-17374661-wctvp\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:32:00.017381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.017051    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"]
Oct 30 01:32:00.025378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.025342    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e92648-nxxfx"]
Oct 30 01:32:00.070058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.069972    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4352416c-a1ef-417c-9052-6274a29fd052-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4352416c-a1ef-417c-9052-6274a29fd052-plugins-config-volume\") pod \"4352416c-a1ef-417c-9052-6274a29fd052\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") "
Oct 30 01:32:00.070252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.070082    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4352416c-a1ef-417c-9052-6274a29fd052-kube-api-access-mbrdz\" (UniqueName: \"kubernetes.io/projected/4352416c-a1ef-417c-9052-6274a29fd052-kube-api-access-mbrdz\") pod \"4352416c-a1ef-417c-9052-6274a29fd052\" (UID: \"4352416c-a1ef-417c-9052-6274a29fd052\") "
Oct 30 01:32:00.070372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.070333    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:00.070444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.070387    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4fh2\" (UniqueName: \"kubernetes.io/projected/abdc9c46-8122-45a4-9080-cba7ab8bd44e-kube-api-access-k4fh2\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:00.070546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.070443    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4352416c-a1ef-417c-9052-6274a29fd052-plugins-config-volume" pod "4352416c-a1ef-417c-9052-6274a29fd052" (UID: "4352416c-a1ef-417c-9052-6274a29fd052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:00.070546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.070524    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:00.070674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.070659    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4352416c-a1ef-417c-9052-6274a29fd052-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:00.070743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.070680    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6d6\" (UniqueName: \"kubernetes.io/projected/c24262f6-4765-4404-86c0-87cbcd13dd97-kube-api-access-tb6d6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:00.071033 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.071010    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:00.072837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.072816    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4352416c-a1ef-417c-9052-6274a29fd052-kube-api-access-mbrdz" pod "4352416c-a1ef-417c-9052-6274a29fd052" (UID: "4352416c-a1ef-417c-9052-6274a29fd052"). InnerVolumeSpecName "kube-api-access-mbrdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:00.072984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.072967    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:00.090071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.090032    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4fh2\" (UniqueName: \"kubernetes.io/projected/abdc9c46-8122-45a4-9080-cba7ab8bd44e-kube-api-access-k4fh2\") pod \"maas-api-revoke-sub-df878348-j9fjh\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:00.157983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.157942    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:32:00.168849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.168824    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:00.171368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.171346    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbrdz\" (UniqueName: \"kubernetes.io/projected/4352416c-a1ef-417c-9052-6274a29fd052-kube-api-access-mbrdz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:00.460951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.460912    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"]
Oct 30 01:32:00.461991 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:32:00.461963    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db7542b_893d_4d17_85e2_ccadaf949201.slice/crio-210d178e6e780b5559a5ba5e7388442a21ea1537715b648e7fe2d605e49b9f30 WatchSource:0}: Error finding container 210d178e6e780b5559a5ba5e7388442a21ea1537715b648e7fe2d605e49b9f30: Status 404 returned error can't find the container with id 210d178e6e780b5559a5ba5e7388442a21ea1537715b648e7fe2d605e49b9f30
Oct 30 01:32:00.474859 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.474826    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"]
Oct 30 01:32:00.476657 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:32:00.476629    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdc9c46_8122_45a4_9080_cba7ab8bd44e.slice/crio-5e1d48060685a3f8a448f4d6a51bb18059ddc11b1f6dde539da2211d377de6a4 WatchSource:0}: Error finding container 5e1d48060685a3f8a448f4d6a51bb18059ddc11b1f6dde539da2211d377de6a4: Status 404 returned error can't find the container with id 5e1d48060685a3f8a448f4d6a51bb18059ddc11b1f6dde539da2211d377de6a4
Oct 30 01:32:00.578973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.578934    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161f57e6-e06e-4fd3-8254-a99af6019ce0" path="/var/lib/kubelet/pods/161f57e6-e06e-4fd3-8254-a99af6019ce0/volumes"
Oct 30 01:32:00.660074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.660030    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp" event={"ID":"6db7542b-893d-4d17-85e2-ccadaf949201","Type":"ContainerStarted","Data":"210d178e6e780b5559a5ba5e7388442a21ea1537715b648e7fe2d605e49b9f30"}
Oct 30 01:32:00.661464 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.661428    5104 generic.go:356] "Generic (PLEG): container finished" podID="c24262f6-4765-4404-86c0-87cbcd13dd97" containerID="768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae" exitCode=0
Oct 30 01:32:00.661547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.661490    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"
Oct 30 01:32:00.661547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.661518    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" event={"ID":"c24262f6-4765-4404-86c0-87cbcd13dd97","Type":"ContainerDied","Data":"768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae"}
Oct 30 01:32:00.661630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.661581    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj" event={"ID":"c24262f6-4765-4404-86c0-87cbcd13dd97","Type":"ContainerDied","Data":"529aef5988073c820203315fc36a23efecb09542fefb7d0435ee8e82380fb20d"}
Oct 30 01:32:00.661630 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.661607    5104 scope.go:122] "RemoveContainer" containerID="768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae"
Oct 30 01:32:00.663187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.663158    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh" event={"ID":"abdc9c46-8122-45a4-9080-cba7ab8bd44e","Type":"ContainerStarted","Data":"5e1d48060685a3f8a448f4d6a51bb18059ddc11b1f6dde539da2211d377de6a4"}
Oct 30 01:32:00.665062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.665039    5104 generic.go:356] "Generic (PLEG): container finished" podID="4352416c-a1ef-417c-9052-6274a29fd052" containerID="43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d" exitCode=0
Oct 30 01:32:00.665185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.665082    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"
Oct 30 01:32:00.665185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.665077    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" event={"ID":"4352416c-a1ef-417c-9052-6274a29fd052","Type":"ContainerDied","Data":"43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d"}
Oct 30 01:32:00.665185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.665139    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt" event={"ID":"4352416c-a1ef-417c-9052-6274a29fd052","Type":"ContainerDied","Data":"9d2f0dfff4a3ff71d73b03b09d7974124d2fd82fbb2f7e63a9433f7987118b76"}
Oct 30 01:32:00.673415 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.673359    5104 scope.go:122] "RemoveContainer" containerID="768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae"
Oct 30 01:32:00.673742 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:00.673717    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae\": container with ID starting with 768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae not found: ID does not exist" containerID="768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae"
Oct 30 01:32:00.673830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.673756    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae"} err="failed to get container status \"768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae\": rpc error: code = NotFound desc = could not find container \"768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae\": container with ID starting with 768bdb592069efcb9167179c8825290ea85137a5a3633a841b33d7ae02ba14ae not found: ID does not exist"
Oct 30 01:32:00.673830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.673784    5104 scope.go:122] "RemoveContainer" containerID="43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d"
Oct 30 01:32:00.699251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.699227    5104 scope.go:122] "RemoveContainer" containerID="43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d"
Oct 30 01:32:00.699647 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:00.699622    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d\": container with ID starting with 43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d not found: ID does not exist" containerID="43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d"
Oct 30 01:32:00.699758 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.699658    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d"} err="failed to get container status \"43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d\": rpc error: code = NotFound desc = could not find container \"43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d\": container with ID starting with 43cfdc12e7c0a5cd60bd85ba6011a95d9ab44c3ba0b9ae20353ed81b55834e0d not found: ID does not exist"
Oct 30 01:32:00.711119 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.711027    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"]
Oct 30 01:32:00.713938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.713906    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d81f0a-8486954d99-6xtgt"]
Oct 30 01:32:00.747852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.747816    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"]
Oct 30 01:32:00.752149 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:00.752116    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d81f0a-f86b6dc76-ffzwj"]
Oct 30 01:32:01.675759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:01.675629    5104 generic.go:356] "Generic (PLEG): container finished" podID="abdc9c46-8122-45a4-9080-cba7ab8bd44e" containerID="772ec177ee16993e9e114c9e1f986cc278d228bc90f6594762e4fd93a52c7d19" exitCode=0
Oct 30 01:32:01.675759 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:01.675679    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh" event={"ID":"abdc9c46-8122-45a4-9080-cba7ab8bd44e","Type":"ContainerDied","Data":"772ec177ee16993e9e114c9e1f986cc278d228bc90f6594762e4fd93a52c7d19"}
Oct 30 01:32:01.678869 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:01.678840    5104 generic.go:356] "Generic (PLEG): container finished" podID="6db7542b-893d-4d17-85e2-ccadaf949201" containerID="1aa8df8b2972462589c6c254ecb48e0d3a382a16cf5443fca5dbe8ad45f40c16" exitCode=0
Oct 30 01:32:01.679022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:01.678885    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp" event={"ID":"6db7542b-893d-4d17-85e2-ccadaf949201","Type":"ContainerDied","Data":"1aa8df8b2972462589c6c254ecb48e0d3a382a16cf5443fca5dbe8ad45f40c16"}
Oct 30 01:32:02.429640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:02.429599    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" podUID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:32:02.429640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:02.429613    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" podUID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:32:02.576854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:02.576815    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4352416c-a1ef-417c-9052-6274a29fd052" path="/var/lib/kubelet/pods/4352416c-a1ef-417c-9052-6274a29fd052/volumes"
Oct 30 01:32:02.577272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:02.577256    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24262f6-4765-4404-86c0-87cbcd13dd97" path="/var/lib/kubelet/pods/c24262f6-4765-4404-86c0-87cbcd13dd97/volumes"
Oct 30 01:32:02.864549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:02.864519    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:32:02.901205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:02.901176    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:03.004312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004222    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-openshift-ca\") pod \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") "
Oct 30 01:32:03.004312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004298    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-xks-ca\") pod \"6db7542b-893d-4d17-85e2-ccadaf949201\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") "
Oct 30 01:32:03.004588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004373    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-xks-ca\") pod \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") "
Oct 30 01:32:03.004588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004419    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db7542b-893d-4d17-85e2-ccadaf949201-kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/6db7542b-893d-4d17-85e2-ccadaf949201-kube-api-access-7czs8\") pod \"6db7542b-893d-4d17-85e2-ccadaf949201\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") "
Oct 30 01:32:03.004588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004509    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-openshift-ca\") pod \"6db7542b-893d-4d17-85e2-ccadaf949201\" (UID: \"6db7542b-893d-4d17-85e2-ccadaf949201\") "
Oct 30 01:32:03.004588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004574    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdc9c46-8122-45a4-9080-cba7ab8bd44e-kube-api-access-k4fh2\" (UniqueName: \"kubernetes.io/projected/abdc9c46-8122-45a4-9080-cba7ab8bd44e-kube-api-access-k4fh2\") pod \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\" (UID: \"abdc9c46-8122-45a4-9080-cba7ab8bd44e\") "
Oct 30 01:32:03.004799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004668    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-openshift-ca" pod "abdc9c46-8122-45a4-9080-cba7ab8bd44e" (UID: "abdc9c46-8122-45a4-9080-cba7ab8bd44e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:03.004989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.004967    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:03.005104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.005076    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-openshift-ca" pod "6db7542b-893d-4d17-85e2-ccadaf949201" (UID: "6db7542b-893d-4d17-85e2-ccadaf949201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:03.006913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.006883    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db7542b-893d-4d17-85e2-ccadaf949201-kube-api-access-7czs8" pod "6db7542b-893d-4d17-85e2-ccadaf949201" (UID: "6db7542b-893d-4d17-85e2-ccadaf949201"). InnerVolumeSpecName "kube-api-access-7czs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:03.007067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.007036    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-xks-ca" pod "abdc9c46-8122-45a4-9080-cba7ab8bd44e" (UID: "abdc9c46-8122-45a4-9080-cba7ab8bd44e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:03.007382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.007349    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdc9c46-8122-45a4-9080-cba7ab8bd44e-kube-api-access-k4fh2" pod "abdc9c46-8122-45a4-9080-cba7ab8bd44e" (UID: "abdc9c46-8122-45a4-9080-cba7ab8bd44e"). InnerVolumeSpecName "kube-api-access-k4fh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:03.007513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.007392    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-xks-ca" pod "6db7542b-893d-4d17-85e2-ccadaf949201" (UID: "6db7542b-893d-4d17-85e2-ccadaf949201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:03.106082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.106037    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdc9c46-8122-45a4-9080-cba7ab8bd44e-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:03.106082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.106077    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czs8\" (UniqueName: \"kubernetes.io/projected/6db7542b-893d-4d17-85e2-ccadaf949201-kube-api-access-7czs8\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:03.106082 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.106092    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:03.106424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.106105    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4fh2\" (UniqueName: \"kubernetes.io/projected/abdc9c46-8122-45a4-9080-cba7ab8bd44e-kube-api-access-k4fh2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:03.106424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.106121    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db7542b-893d-4d17-85e2-ccadaf949201-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:03.369806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.369653    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:32:03.696858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.696813    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"
Oct 30 01:32:03.696858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.696836    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp" event={"ID":"6db7542b-893d-4d17-85e2-ccadaf949201","Type":"ContainerDied","Data":"210d178e6e780b5559a5ba5e7388442a21ea1537715b648e7fe2d605e49b9f30"}
Oct 30 01:32:03.697171 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.696892    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210d178e6e780b5559a5ba5e7388442a21ea1537715b648e7fe2d605e49b9f30"
Oct 30 01:32:03.701061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.701035    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"
Oct 30 01:32:03.701223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.701060    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh" event={"ID":"abdc9c46-8122-45a4-9080-cba7ab8bd44e","Type":"ContainerDied","Data":"5e1d48060685a3f8a448f4d6a51bb18059ddc11b1f6dde539da2211d377de6a4"}
Oct 30 01:32:03.701223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:03.701117    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e1d48060685a3f8a448f4d6a51bb18059ddc11b1f6dde539da2211d377de6a4"
Oct 30 01:32:07.202255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.202215    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"]
Oct 30 01:32:07.202790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.202749    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy" containerID="cri-o://ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a" gracePeriod=30
Oct 30 01:32:07.226752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.226709    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"]
Oct 30 01:32:07.227220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.227182    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy" containerID="cri-o://cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced" gracePeriod=30
Oct 30 01:32:07.229651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.229624    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"]
Oct 30 01:32:07.229866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.229843    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy" containerID="cri-o://c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667" gracePeriod=30
Oct 30 01:32:07.235931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.235899    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"]
Oct 30 01:32:07.236150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.236126    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy" containerID="cri-o://49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8" gracePeriod=30
Oct 30 01:32:07.824811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.824764    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"]
Oct 30 01:32:07.825254 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825240    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c24262f6-4765-4404-86c0-87cbcd13dd97" containerName="payload-processing"
Oct 30 01:32:07.825331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825255    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24262f6-4765-4404-86c0-87cbcd13dd97" containerName="payload-processing"
Oct 30 01:32:07.825331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825289    5104 cpu_manager.go:414] "containerMap: removing container" podUID="abdc9c46-8122-45a4-9080-cba7ab8bd44e" containerName="revoke-keys"
Oct 30 01:32:07.825331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825295    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdc9c46-8122-45a4-9080-cba7ab8bd44e" containerName="revoke-keys"
Oct 30 01:32:07.825331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825306    5104 cpu_manager.go:414] "containerMap: removing container" podUID="6db7542b-893d-4d17-85e2-ccadaf949201" containerName="revoke-keys"
Oct 30 01:32:07.825331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825312    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db7542b-893d-4d17-85e2-ccadaf949201" containerName="revoke-keys"
Oct 30 01:32:07.825331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825322    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4352416c-a1ef-417c-9052-6274a29fd052" containerName="payload-pre-processing"
Oct 30 01:32:07.825331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825327    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4352416c-a1ef-417c-9052-6274a29fd052" containerName="payload-pre-processing"
Oct 30 01:32:07.825733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825421    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdc9c46-8122-45a4-9080-cba7ab8bd44e" containerName="revoke-keys"
Oct 30 01:32:07.825733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825430    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db7542b-893d-4d17-85e2-ccadaf949201" containerName="revoke-keys"
Oct 30 01:32:07.825733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825438    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4352416c-a1ef-417c-9052-6274a29fd052" containerName="payload-pre-processing"
Oct 30 01:32:07.825733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.825447    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24262f6-4765-4404-86c0-87cbcd13dd97" containerName="payload-processing"
Oct 30 01:32:07.828843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.828819    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:07.846130 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.846103    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:32:07.846276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.846182    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:32:07.879506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.861997    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"]
Oct 30 01:32:07.884846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.884810    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:32:07.885025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.884942    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:32:07.979292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.979255    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:07.979497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.979300    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:07.979497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:07.979330    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlc79\" (UniqueName: \"kubernetes.io/projected/3eab6cbc-6c03-4c00-9a89-97386b957728-kube-api-access-xlc79\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.081084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.080992    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.081084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.081046    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.081084 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.081075    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlc79\" (UniqueName: \"kubernetes.io/projected/3eab6cbc-6c03-4c00-9a89-97386b957728-kube-api-access-xlc79\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.081810 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.081770    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.083714 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.083669    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.100410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.100379    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlc79\" (UniqueName: \"kubernetes.io/projected/3eab6cbc-6c03-4c00-9a89-97386b957728-kube-api-access-xlc79\") pod \"maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.143231 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.143191    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:08.439533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.439486    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"]
Oct 30 01:32:08.442156 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:32:08.442126    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eab6cbc_6c03_4c00_9a89_97386b957728.slice/crio-ef7529f198877b858208c36ab5fdcd445b9bec13a23e52accb544e5a8d0ce02f WatchSource:0}: Error finding container ef7529f198877b858208c36ab5fdcd445b9bec13a23e52accb544e5a8d0ce02f: Status 404 returned error can't find the container with id ef7529f198877b858208c36ab5fdcd445b9bec13a23e52accb544e5a8d0ce02f
Oct 30 01:32:08.741191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:08.741158    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8" event={"ID":"3eab6cbc-6c03-4c00-9a89-97386b957728","Type":"ContainerStarted","Data":"ef7529f198877b858208c36ab5fdcd445b9bec13a23e52accb544e5a8d0ce02f"}
Oct 30 01:32:09.103606 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:09.103510    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:32:09.750677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:09.750595    5104 generic.go:356] "Generic (PLEG): container finished" podID="3eab6cbc-6c03-4c00-9a89-97386b957728" containerID="a884d62b66d27200fe9e234a21804a78bf58c5dcf07ddc0ec15194fdb95ac04d" exitCode=0
Oct 30 01:32:09.750677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:09.750643    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8" event={"ID":"3eab6cbc-6c03-4c00-9a89-97386b957728","Type":"ContainerDied","Data":"a884d62b66d27200fe9e234a21804a78bf58c5dcf07ddc0ec15194fdb95ac04d"}
Oct 30 01:32:10.918291 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:10.918263    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:11.115319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.115227    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-openshift-ca\") pod \"3eab6cbc-6c03-4c00-9a89-97386b957728\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") "
Oct 30 01:32:11.115527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.115361    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-xks-ca\") pod \"3eab6cbc-6c03-4c00-9a89-97386b957728\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") "
Oct 30 01:32:11.115527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.115386    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eab6cbc-6c03-4c00-9a89-97386b957728-kube-api-access-xlc79\" (UniqueName: \"kubernetes.io/projected/3eab6cbc-6c03-4c00-9a89-97386b957728-kube-api-access-xlc79\") pod \"3eab6cbc-6c03-4c00-9a89-97386b957728\" (UID: \"3eab6cbc-6c03-4c00-9a89-97386b957728\") "
Oct 30 01:32:11.115742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.115697    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-openshift-ca" pod "3eab6cbc-6c03-4c00-9a89-97386b957728" (UID: "3eab6cbc-6c03-4c00-9a89-97386b957728"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:11.115890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.115847    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:11.117767 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.117741    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-xks-ca" pod "3eab6cbc-6c03-4c00-9a89-97386b957728" (UID: "3eab6cbc-6c03-4c00-9a89-97386b957728"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:11.117913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.117788    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eab6cbc-6c03-4c00-9a89-97386b957728-kube-api-access-xlc79" pod "3eab6cbc-6c03-4c00-9a89-97386b957728" (UID: "3eab6cbc-6c03-4c00-9a89-97386b957728"). InnerVolumeSpecName "kube-api-access-xlc79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:11.216815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.216779    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab6cbc-6c03-4c00-9a89-97386b957728-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:11.216815 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.216809    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlc79\" (UniqueName: \"kubernetes.io/projected/3eab6cbc-6c03-4c00-9a89-97386b957728-kube-api-access-xlc79\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:11.737535 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.737500    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:32:11.767205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.767176    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"
Oct 30 01:32:11.767378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.767219    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8" event={"ID":"3eab6cbc-6c03-4c00-9a89-97386b957728","Type":"ContainerDied","Data":"ef7529f198877b858208c36ab5fdcd445b9bec13a23e52accb544e5a8d0ce02f"}
Oct 30 01:32:11.767378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:11.767276    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef7529f198877b858208c36ab5fdcd445b9bec13a23e52accb544e5a8d0ce02f"
Oct 30 01:32:12.429615 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.429554    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" podUID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:32:12.430406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.430342    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" podUID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:32:12.521606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.521575    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:32:12.529785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.529752    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79636883-5bc5-478e-a99f-50df03195498-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79636883-5bc5-478e-a99f-50df03195498-istio-podinfo\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.529959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.529827    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-data\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.529959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.529862    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-certs\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.529959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.529907    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-credential-socket\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.529959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.529957    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-kube-api-access-w5jdm\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-kube-api-access-w5jdm\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.530196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.530021    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-envoy\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.530196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.530118    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-istio-token\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-istio-token\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.530196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.530190    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-socket\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.530380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.530243    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79636883-5bc5-478e-a99f-50df03195498-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79636883-5bc5-478e-a99f-50df03195498-istiod-ca-cert\") pod \"79636883-5bc5-478e-a99f-50df03195498\" (UID: \"79636883-5bc5-478e-a99f-50df03195498\") "
Oct 30 01:32:12.531281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.531007    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79636883-5bc5-478e-a99f-50df03195498-istiod-ca-cert" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:12.531558 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.531519    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-socket" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.531824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.531675    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-data" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.531824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.531756    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-certs" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.532109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.532064    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-credential-socket" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.533122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.533097    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-envoy" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.534179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.534151    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-istio-token" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.534774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.534747    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-kube-api-access-w5jdm" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "kube-api-access-w5jdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.535249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.535215    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79636883-5bc5-478e-a99f-50df03195498-istio-podinfo" pod "79636883-5bc5-478e-a99f-50df03195498" (UID: "79636883-5bc5-478e-a99f-50df03195498"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:32:12.602066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.602036    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:32:12.608887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.608856    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:32:12.614401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.614372    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:32:12.632018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.631953    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-credential-socket\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.632018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.631998    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-token\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-token\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632027    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-data\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632055    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-socket\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632095    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-podinfo\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632134    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-socket\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632193    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-podinfo\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632224    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-credential-socket\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632260    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-token\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-token\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632267    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-credential-socket" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632291    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a0bc92-3340-43a0-811c-db63789c6a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a0bc92-3340-43a0-811c-db63789c6a05-istiod-ca-cert\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632338    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-credential-socket\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632375    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-kube-api-access-tqbfm\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-kube-api-access-tqbfm\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632409    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-token\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-token\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632448    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istiod-ca-cert\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632507    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-kube-api-access-bzxwj\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-kube-api-access-bzxwj\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632549    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-socket\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632586    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-podinfo\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632634    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-envoy\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632681    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-certs\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632716    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b35fbd-906b-450d-99a0-b5b0f405b607-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b35fbd-906b-450d-99a0-b5b0f405b607-istiod-ca-cert\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632749    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-kube-api-access-nbpxb\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-kube-api-access-nbpxb\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.632793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632788    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-certs\") pod \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\" (UID: \"4c1bf6a4-4e23-46b6-bab8-74f806685b90\") "
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632823    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-certs\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632876    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-data\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632910    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-data\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632958    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-envoy\") pod \"f8b35fbd-906b-450d-99a0-b5b0f405b607\" (UID: \"f8b35fbd-906b-450d-99a0-b5b0f405b607\") "
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633020    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-envoy\") pod \"c5a0bc92-3340-43a0-811c-db63789c6a05\" (UID: \"c5a0bc92-3340-43a0-811c-db63789c6a05\") "
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633439    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79636883-5bc5-478e-a99f-50df03195498-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633473    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633488    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633502    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633516    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5jdm\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-kube-api-access-w5jdm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633532    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633546    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633559    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79636883-5bc5-478e-a99f-50df03195498-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.633574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633572    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79636883-5bc5-478e-a99f-50df03195498-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.634322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.633586    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79636883-5bc5-478e-a99f-50df03195498-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.638208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632822    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-credential-socket" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.638208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.632995    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-socket" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.635137    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-credential-socket" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.635503    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a0bc92-3340-43a0-811c-db63789c6a05-istiod-ca-cert" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.635793    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-socket" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.637092    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istiod-ca-cert" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.637131    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-certs" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.637162    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-data" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.637215    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-podinfo" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.637712    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b35fbd-906b-450d-99a0-b5b0f405b607-istiod-ca-cert" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.637916    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-certs" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.637967    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-podinfo" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:32:12.638407 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.638180    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-data" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.639036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.638498    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-data" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.639036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.638523    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-socket" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.639036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.638730    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-certs" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.639036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.638798    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-envoy" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.640398 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.640372    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-envoy" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.642567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.642543    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-envoy" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:12.643799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.643772    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-podinfo" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:32:12.643799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.643786    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-token" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.643970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.643819    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-token" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.643970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.643842    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-kube-api-access-tqbfm" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "kube-api-access-tqbfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.643970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.643909    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-kube-api-access-bzxwj" pod "4c1bf6a4-4e23-46b6-bab8-74f806685b90" (UID: "4c1bf6a4-4e23-46b6-bab8-74f806685b90"). InnerVolumeSpecName "kube-api-access-bzxwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.645036 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.645016    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-kube-api-access-nbpxb" pod "f8b35fbd-906b-450d-99a0-b5b0f405b607" (UID: "f8b35fbd-906b-450d-99a0-b5b0f405b607"). InnerVolumeSpecName "kube-api-access-nbpxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.645904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.645879    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-token" pod "c5a0bc92-3340-43a0-811c-db63789c6a05" (UID: "c5a0bc92-3340-43a0-811c-db63789c6a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:12.673261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.673229    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"]
Oct 30 01:32:12.679806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.679773    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"]
Oct 30 01:32:12.735161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735117    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735161 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735160    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735178    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735195    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a0bc92-3340-43a0-811c-db63789c6a05-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735209    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735223    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbfm\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-kube-api-access-tqbfm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735243    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735257    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735271    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzxwj\" (UniqueName: \"kubernetes.io/projected/4c1bf6a4-4e23-46b6-bab8-74f806685b90-kube-api-access-bzxwj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735285    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735301    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735317    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735330    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735343    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b35fbd-906b-450d-99a0-b5b0f405b607-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735357    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbpxb\" (UniqueName: \"kubernetes.io/projected/f8b35fbd-906b-450d-99a0-b5b0f405b607-kube-api-access-nbpxb\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735371    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735386    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735399    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735414    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735428    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735442    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735471    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735485    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735496    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1bf6a4-4e23-46b6-bab8-74f806685b90-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735505    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a0bc92-3340-43a0-811c-db63789c6a05-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.735795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.735513    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b35fbd-906b-450d-99a0-b5b0f405b607-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:12.778588 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.778500    5104 generic.go:356] "Generic (PLEG): container finished" podID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerID="49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8" exitCode=0
Oct 30 01:32:12.778763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.778589    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"
Oct 30 01:32:12.778763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.778623    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" event={"ID":"f8b35fbd-906b-450d-99a0-b5b0f405b607","Type":"ContainerDied","Data":"49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8"}
Oct 30 01:32:12.778763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.778679    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v" event={"ID":"f8b35fbd-906b-450d-99a0-b5b0f405b607","Type":"ContainerDied","Data":"b89807a1d7fe9a68b162fb387c23b23124bab0464a683ec566d4f2599b0374f0"}
Oct 30 01:32:12.778763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.778711    5104 scope.go:122] "RemoveContainer" containerID="49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8"
Oct 30 01:32:12.784018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.783737    5104 generic.go:356] "Generic (PLEG): container finished" podID="79636883-5bc5-478e-a99f-50df03195498" containerID="ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a" exitCode=0
Oct 30 01:32:12.784018 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.783896    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" event={"ID":"79636883-5bc5-478e-a99f-50df03195498","Type":"ContainerDied","Data":"ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a"}
Oct 30 01:32:12.784885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.784296    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"
Oct 30 01:32:12.789478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.787550    5104 generic.go:356] "Generic (PLEG): container finished" podID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerID="c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667" exitCode=0
Oct 30 01:32:12.789478 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.787778    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"
Oct 30 01:32:12.792247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.791754    5104 generic.go:356] "Generic (PLEG): container finished" podID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerID="cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced" exitCode=0
Oct 30 01:32:12.792247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.792059    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"
Oct 30 01:32:12.792604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.792569    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5" event={"ID":"79636883-5bc5-478e-a99f-50df03195498","Type":"ContainerDied","Data":"d6dab1f78146281d2338ef6ed3ce6891617d42e4e615ce1ab5b4bd5ca77c8ee0"}
Oct 30 01:32:12.792679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.792642    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" event={"ID":"4c1bf6a4-4e23-46b6-bab8-74f806685b90","Type":"ContainerDied","Data":"c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667"}
Oct 30 01:32:12.792745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.792685    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg" event={"ID":"4c1bf6a4-4e23-46b6-bab8-74f806685b90","Type":"ContainerDied","Data":"b571be44251caa17d8ed67386c855d5f2e94a4f58215d061764265345609d7ee"}
Oct 30 01:32:12.792745 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.792719    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" event={"ID":"c5a0bc92-3340-43a0-811c-db63789c6a05","Type":"ContainerDied","Data":"cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced"}
Oct 30 01:32:12.792852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.792750    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g" event={"ID":"c5a0bc92-3340-43a0-811c-db63789c6a05","Type":"ContainerDied","Data":"8c6b1cb5973cce72c005f0f24badcbf4e2c1902296ded31f3c82d0d0779f578e"}
Oct 30 01:32:12.822713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.821389    5104 scope.go:122] "RemoveContainer" containerID="9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d"
Oct 30 01:32:12.834970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.834945    5104 scope.go:122] "RemoveContainer" containerID="49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8"
Oct 30 01:32:12.835313 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:12.835289    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8\": container with ID starting with 49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8 not found: ID does not exist" containerID="49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8"
Oct 30 01:32:12.835375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.835323    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8"} err="failed to get container status \"49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8\": rpc error: code = NotFound desc = could not find container \"49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8\": container with ID starting with 49a5e7989adf077f9e00b5058bef60160e8d135f18b95b5d733e59bba9c73de8 not found: ID does not exist"
Oct 30 01:32:12.835375 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.835341    5104 scope.go:122] "RemoveContainer" containerID="9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d"
Oct 30 01:32:12.835646 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:12.835626    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d\": container with ID starting with 9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d not found: ID does not exist" containerID="9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d"
Oct 30 01:32:12.835705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.835653    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d"} err="failed to get container status \"9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d\": rpc error: code = NotFound desc = could not find container \"9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d\": container with ID starting with 9ba1025433075fb115c9b367f8a57822054d4034b8a4330748596e6afa2e9a6d not found: ID does not exist"
Oct 30 01:32:12.835705 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.835669    5104 scope.go:122] "RemoveContainer" containerID="ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a"
Oct 30 01:32:12.846561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.846532    5104 scope.go:122] "RemoveContainer" containerID="ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a"
Oct 30 01:32:12.846877 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:12.846853    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a\": container with ID starting with ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a not found: ID does not exist" containerID="ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a"
Oct 30 01:32:12.846972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.846886    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a"} err="failed to get container status \"ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a\": rpc error: code = NotFound desc = could not find container \"ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a\": container with ID starting with ba6cbda43c0c9b87cee7b2c3c96533e7f9443b15be416425edee38d673851d8a not found: ID does not exist"
Oct 30 01:32:12.846972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.846905    5104 scope.go:122] "RemoveContainer" containerID="c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667"
Oct 30 01:32:12.857188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.857166    5104 scope.go:122] "RemoveContainer" containerID="c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667"
Oct 30 01:32:12.857510 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:12.857485    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667\": container with ID starting with c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667 not found: ID does not exist" containerID="c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667"
Oct 30 01:32:12.857579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.857519    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667"} err="failed to get container status \"c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667\": rpc error: code = NotFound desc = could not find container \"c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667\": container with ID starting with c269ee50dbce558b841d8ff253b484adfac2750cca9b52c1a3429ad500991667 not found: ID does not exist"
Oct 30 01:32:12.857579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.857536    5104 scope.go:122] "RemoveContainer" containerID="cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced"
Oct 30 01:32:12.867624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.867602    5104 scope.go:122] "RemoveContainer" containerID="cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced"
Oct 30 01:32:12.867935 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:12.867914    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced\": container with ID starting with cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced not found: ID does not exist" containerID="cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced"
Oct 30 01:32:12.867989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.867943    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced"} err="failed to get container status \"cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced\": rpc error: code = NotFound desc = could not find container \"cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced\": container with ID starting with cd7f367a8b10aeb0c4294381771e9bfd9e3a65f60097b4b81e4734b2d49d9ced not found: ID does not exist"
Oct 30 01:32:12.871882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.871854    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"]
Oct 30 01:32:12.877939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.877907    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-fc86v"]
Oct 30 01:32:12.908464 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.908427    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"]
Oct 30 01:32:12.914311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.914258    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-nbf6g"]
Oct 30 01:32:12.944938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.944905    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"]
Oct 30 01:32:12.947878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.947852    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zzzwg"]
Oct 30 01:32:12.982471 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.982432    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"]
Oct 30 01:32:12.989031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:12.988999    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-5m8k5"]
Oct 30 01:32:14.580442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:14.580408    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" path="/var/lib/kubelet/pods/4c1bf6a4-4e23-46b6-bab8-74f806685b90/volumes"
Oct 30 01:32:14.580917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:14.580901    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79636883-5bc5-478e-a99f-50df03195498" path="/var/lib/kubelet/pods/79636883-5bc5-478e-a99f-50df03195498/volumes"
Oct 30 01:32:14.581307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:14.581294    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" path="/var/lib/kubelet/pods/c5a0bc92-3340-43a0-811c-db63789c6a05/volumes"
Oct 30 01:32:14.582056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:14.582040    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" path="/var/lib/kubelet/pods/f8b35fbd-906b-450d-99a0-b5b0f405b607/volumes"
Oct 30 01:32:17.344708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:17.344669    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"]
Oct 30 01:32:17.356108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:17.356048    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82b647-8p279"]
Oct 30 01:32:18.315508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:18.315473    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"]
Oct 30 01:32:18.585226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:18.585138    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc3bf0c6-928f-4886-bcde-3805f4ee82b8" path="/var/lib/kubelet/pods/cc3bf0c6-928f-4886-bcde-3805f4ee82b8/volumes"
Oct 30 01:32:20.176923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.176888    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 01:32:20.207617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.207591    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-kube-api-access-7dvfg\" (UniqueName: \"kubernetes.io/projected/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-kube-api-access-7dvfg\") pod \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") "
Oct 30 01:32:20.207770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.207638    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-plugins-config-volume\") pod \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\" (UID: \"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0\") "
Oct 30 01:32:20.208076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.208051    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-plugins-config-volume" pod "ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" (UID: "ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:20.209920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.209892    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-kube-api-access-7dvfg" pod "ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" (UID: "ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0"). InnerVolumeSpecName "kube-api-access-7dvfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:20.224568 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.224539    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 01:32:20.308644 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.308608    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-kube-api-access-rx2hc\" (UniqueName: \"kubernetes.io/projected/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-kube-api-access-rx2hc\") pod \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") "
Oct 30 01:32:20.308876 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.308704    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-plugins-config-volume\") pod \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\" (UID: \"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1\") "
Oct 30 01:32:20.309057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.309032    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:20.309143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.309061    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvfg\" (UniqueName: \"kubernetes.io/projected/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0-kube-api-access-7dvfg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:20.309388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.309359    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-plugins-config-volume" pod "2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" (UID: "2c7427cc-2a6f-4b80-abaa-f41f3be1fea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:20.310971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.310948    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-kube-api-access-rx2hc" pod "2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" (UID: "2c7427cc-2a6f-4b80-abaa-f41f3be1fea1"). InnerVolumeSpecName "kube-api-access-rx2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:20.409822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.409737    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:20.409822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.409771    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2hc\" (UniqueName: \"kubernetes.io/projected/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1-kube-api-access-rx2hc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:20.728490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.728350    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"]
Oct 30 01:32:20.728959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.728935    5104 cpu_manager.go:414] "containerMap: removing container" podUID="3eab6cbc-6c03-4c00-9a89-97386b957728" containerName="revoke-keys"
Oct 30 01:32:20.729067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.728962    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eab6cbc-6c03-4c00-9a89-97386b957728" containerName="revoke-keys"
Oct 30 01:32:20.729067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.728987    5104 cpu_manager.go:414] "containerMap: removing container" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy"
Oct 30 01:32:20.729067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.728998    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy"
Oct 30 01:32:20.729067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729008    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy"
Oct 30 01:32:20.729067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729016    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy"
Oct 30 01:32:20.729067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729032    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy"
Oct 30 01:32:20.729067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729040    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729071    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729079    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729101    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" containerName="payload-processing"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729109    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" containerName="payload-processing"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729131    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729139    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729150    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" containerName="payload-pre-processing"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729159    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" containerName="payload-pre-processing"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729270    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eab6cbc-6c03-4c00-9a89-97386b957728" containerName="revoke-keys"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729287    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="79636883-5bc5-478e-a99f-50df03195498" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729297    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a0bc92-3340-43a0-811c-db63789c6a05" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729308    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1bf6a4-4e23-46b6-bab8-74f806685b90" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729322    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729334    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" containerName="payload-processing"
Oct 30 01:32:20.729488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.729345    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" containerName="payload-pre-processing"
Oct 30 01:32:20.733256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.733233    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.746833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.746798    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"]
Oct 30 01:32:20.747333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.747319    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b35fbd-906b-450d-99a0-b5b0f405b607" containerName="istio-proxy"
Oct 30 01:32:20.749300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.749278    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:32:20.750675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.750659    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.751984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.751958    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"]
Oct 30 01:32:20.754786 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.754757    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:32:20.760691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.760664    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:32:20.772075 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.772042    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"]
Oct 30 01:32:20.813891 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.813851    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzd2\" (UniqueName: \"kubernetes.io/projected/7b50207f-830b-4526-93b0-001008700eda-kube-api-access-rqzd2\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.814103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.813908    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b50207f-830b-4526-93b0-001008700eda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.814103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.814011    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.814199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.814113    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9n4w\" (UniqueName: \"kubernetes.io/projected/54a3b34a-572b-4c88-b459-14d860ed649b-kube-api-access-q9n4w\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.814199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.814192    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.814319 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.814300    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b50207f-830b-4526-93b0-001008700eda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.849746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.849718    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:32:20.860028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.859997    5104 generic.go:356] "Generic (PLEG): container finished" podID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" containerID="3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996" exitCode=0
Oct 30 01:32:20.860209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.860040    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"
Oct 30 01:32:20.860209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.860082    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" event={"ID":"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1","Type":"ContainerDied","Data":"3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996"}
Oct 30 01:32:20.860209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.860134    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r" event={"ID":"2c7427cc-2a6f-4b80-abaa-f41f3be1fea1","Type":"ContainerDied","Data":"6ed6a9fcead33b3979adbb98fa4dadcb95f7d1fa509e3fbcc815893cc156e59e"}
Oct 30 01:32:20.860209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.860156    5104 scope.go:122] "RemoveContainer" containerID="3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996"
Oct 30 01:32:20.861733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.861704    5104 generic.go:356] "Generic (PLEG): container finished" podID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" containerID="d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311" exitCode=0
Oct 30 01:32:20.861887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.861750    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" event={"ID":"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0","Type":"ContainerDied","Data":"d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311"}
Oct 30 01:32:20.861887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.861760    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"
Oct 30 01:32:20.861887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.861786    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s" event={"ID":"ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0","Type":"ContainerDied","Data":"93e727a74588e9db427bbf6c97405f7ff6c560f201b8848ff20c8dd1d9b3b61e"}
Oct 30 01:32:20.870515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.870496    5104 scope.go:122] "RemoveContainer" containerID="3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996"
Oct 30 01:32:20.870824 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:20.870804    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996\": container with ID starting with 3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996 not found: ID does not exist" containerID="3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996"
Oct 30 01:32:20.870916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.870838    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996"} err="failed to get container status \"3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996\": rpc error: code = NotFound desc = could not find container \"3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996\": container with ID starting with 3b16b4a582acfce34e48abcd39fb1f5530f767c729fd5daa7017366586b20996 not found: ID does not exist"
Oct 30 01:32:20.870916 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.870867    5104 scope.go:122] "RemoveContainer" containerID="d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311"
Oct 30 01:32:20.880593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.880572    5104 scope.go:122] "RemoveContainer" containerID="d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311"
Oct 30 01:32:20.880939 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:20.880916    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311\": container with ID starting with d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311 not found: ID does not exist" containerID="d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311"
Oct 30 01:32:20.881072 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.880949    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311"} err="failed to get container status \"d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311\": rpc error: code = NotFound desc = could not find container \"d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311\": container with ID starting with d4235e2a34a1ea1cb4ede15613b7b71444d540f50a0922847735ba6f5617b311 not found: ID does not exist"
Oct 30 01:32:20.908672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.908590    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"]
Oct 30 01:32:20.916765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.916015    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9n4w\" (UniqueName: \"kubernetes.io/projected/54a3b34a-572b-4c88-b459-14d860ed649b-kube-api-access-q9n4w\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.916765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.916141    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.916765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.916199    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b50207f-830b-4526-93b0-001008700eda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.916765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.916236    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzd2\" (UniqueName: \"kubernetes.io/projected/7b50207f-830b-4526-93b0-001008700eda-kube-api-access-rqzd2\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.916765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.916270    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b50207f-830b-4526-93b0-001008700eda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.916765 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.916298    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.917279 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.917132    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b50207f-830b-4526-93b0-001008700eda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.917806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.917780    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82b647-546f5847f-d9c8r"]
Oct 30 01:32:20.917929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.917830    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.918996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.918969    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.918996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.918989    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b50207f-830b-4526-93b0-001008700eda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.935242 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.935213    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzd2\" (UniqueName: \"kubernetes.io/projected/7b50207f-830b-4526-93b0-001008700eda-kube-api-access-rqzd2\") pod \"maas-api-revoke-sub-3be992ab-gb6lb\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:20.935413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.935388    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9n4w\" (UniqueName: \"kubernetes.io/projected/54a3b34a-572b-4c88-b459-14d860ed649b-kube-api-access-q9n4w\") pod \"maas-api-revoke-sub-b09c2efb-8p7tq\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:20.947324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.947291    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"]
Oct 30 01:32:20.949940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:20.949912    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82b647-7b45bddbb6-xq44s"]
Oct 30 01:32:21.047209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.047115    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:21.066831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.066791    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:21.351492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.351451    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"]
Oct 30 01:32:21.354207 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:32:21.354175    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a3b34a_572b_4c88_b459_14d860ed649b.slice/crio-ef1953d5784049e9daafd669fb42c2534b588b678fce715091d7dee58eea65fa WatchSource:0}: Error finding container ef1953d5784049e9daafd669fb42c2534b588b678fce715091d7dee58eea65fa: Status 404 returned error can't find the container with id ef1953d5784049e9daafd669fb42c2534b588b678fce715091d7dee58eea65fa
Oct 30 01:32:21.380783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.380755    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"]
Oct 30 01:32:21.384775 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:32:21.384750    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b50207f_830b_4526_93b0_001008700eda.slice/crio-596d5dcc5c82b9883cc1a0350c83649731b0ca02e97ea7a536fa160a32dd3403 WatchSource:0}: Error finding container 596d5dcc5c82b9883cc1a0350c83649731b0ca02e97ea7a536fa160a32dd3403: Status 404 returned error can't find the container with id 596d5dcc5c82b9883cc1a0350c83649731b0ca02e97ea7a536fa160a32dd3403
Oct 30 01:32:21.536580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.536526    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:32:21.870902 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.870862    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq" event={"ID":"54a3b34a-572b-4c88-b459-14d860ed649b","Type":"ContainerStarted","Data":"441d837c4f8cad6c2f8a8be17a7bcbefe95a88c5f250f38368048112bd768d80"}
Oct 30 01:32:21.871087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.870928    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq" event={"ID":"54a3b34a-572b-4c88-b459-14d860ed649b","Type":"ContainerStarted","Data":"ef1953d5784049e9daafd669fb42c2534b588b678fce715091d7dee58eea65fa"}
Oct 30 01:32:21.876503 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.876449    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb" event={"ID":"7b50207f-830b-4526-93b0-001008700eda","Type":"ContainerStarted","Data":"a7b772466eec5f44a7dbec3e68a303fda66b6422a556d632b8f20325d8727a8a"}
Oct 30 01:32:21.876775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.876534    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb" event={"ID":"7b50207f-830b-4526-93b0-001008700eda","Type":"ContainerStarted","Data":"596d5dcc5c82b9883cc1a0350c83649731b0ca02e97ea7a536fa160a32dd3403"}
Oct 30 01:32:21.908968 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.908905    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq" podStartSLOduration=1.908884174 podStartE2EDuration="1.908884174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:21.907212749 +0000 UTC m=+2983.454903304" watchObservedRunningTime="2026-10-30 01:32:21.908884174 +0000 UTC m=+2983.456574671"
Oct 30 01:32:21.944045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:21.943938    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb" podStartSLOduration=1.943922903 podStartE2EDuration="1.943922903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:21.942588036 +0000 UTC m=+2983.490278532" watchObservedRunningTime="2026-10-30 01:32:21.943922903 +0000 UTC m=+2983.491613398"
Oct 30 01:32:22.576166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:22.576082    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7427cc-2a6f-4b80-abaa-f41f3be1fea1" path="/var/lib/kubelet/pods/2c7427cc-2a6f-4b80-abaa-f41f3be1fea1/volumes"
Oct 30 01:32:22.576590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:22.576550    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0" path="/var/lib/kubelet/pods/ad6c1f0a-bd07-4b25-ab2b-278b8d6c47d0/volumes"
Oct 30 01:32:22.899560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:22.899445    5104 generic.go:356] "Generic (PLEG): container finished" podID="54a3b34a-572b-4c88-b459-14d860ed649b" containerID="441d837c4f8cad6c2f8a8be17a7bcbefe95a88c5f250f38368048112bd768d80" exitCode=0
Oct 30 01:32:22.899560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:22.899517    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq" event={"ID":"54a3b34a-572b-4c88-b459-14d860ed649b","Type":"ContainerDied","Data":"441d837c4f8cad6c2f8a8be17a7bcbefe95a88c5f250f38368048112bd768d80"}
Oct 30 01:32:22.901989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:22.901963    5104 generic.go:356] "Generic (PLEG): container finished" podID="7b50207f-830b-4526-93b0-001008700eda" containerID="a7b772466eec5f44a7dbec3e68a303fda66b6422a556d632b8f20325d8727a8a" exitCode=0
Oct 30 01:32:22.902190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:22.902029    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb" event={"ID":"7b50207f-830b-4526-93b0-001008700eda","Type":"ContainerDied","Data":"a7b772466eec5f44a7dbec3e68a303fda66b6422a556d632b8f20325d8727a8a"}
Oct 30 01:32:23.316479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:23.316403    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" podUID="4e2470c5-091d-4c27-86b4-f096085566ca" containerName="payload-pre-processing" containerID="cri-o://924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632" gracePeriod=25
Oct 30 01:32:24.082234 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.082208    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:24.084321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.084300    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:24.148402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.148357    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-xks-ca\") pod \"54a3b34a-572b-4c88-b459-14d860ed649b\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") "
Oct 30 01:32:24.148610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.148483    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a3b34a-572b-4c88-b459-14d860ed649b-kube-api-access-q9n4w\" (UniqueName: \"kubernetes.io/projected/54a3b34a-572b-4c88-b459-14d860ed649b-kube-api-access-q9n4w\") pod \"54a3b34a-572b-4c88-b459-14d860ed649b\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") "
Oct 30 01:32:24.148610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.148527    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b50207f-830b-4526-93b0-001008700eda-kube-api-access-rqzd2\" (UniqueName: \"kubernetes.io/projected/7b50207f-830b-4526-93b0-001008700eda-kube-api-access-rqzd2\") pod \"7b50207f-830b-4526-93b0-001008700eda\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") "
Oct 30 01:32:24.148610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.148556    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b50207f-830b-4526-93b0-001008700eda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b50207f-830b-4526-93b0-001008700eda-maas-api-xks-ca\") pod \"7b50207f-830b-4526-93b0-001008700eda\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") "
Oct 30 01:32:24.148845 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.148816    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-openshift-ca\") pod \"54a3b34a-572b-4c88-b459-14d860ed649b\" (UID: \"54a3b34a-572b-4c88-b459-14d860ed649b\") "
Oct 30 01:32:24.148940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.148907    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b50207f-830b-4526-93b0-001008700eda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b50207f-830b-4526-93b0-001008700eda-maas-api-openshift-ca\") pod \"7b50207f-830b-4526-93b0-001008700eda\" (UID: \"7b50207f-830b-4526-93b0-001008700eda\") "
Oct 30 01:32:24.149425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.149310    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-openshift-ca" pod "54a3b34a-572b-4c88-b459-14d860ed649b" (UID: "54a3b34a-572b-4c88-b459-14d860ed649b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:24.149684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.149651    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b50207f-830b-4526-93b0-001008700eda-maas-api-openshift-ca" pod "7b50207f-830b-4526-93b0-001008700eda" (UID: "7b50207f-830b-4526-93b0-001008700eda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:24.150985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.150957    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b50207f-830b-4526-93b0-001008700eda-maas-api-xks-ca" pod "7b50207f-830b-4526-93b0-001008700eda" (UID: "7b50207f-830b-4526-93b0-001008700eda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:24.152493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.151492    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-xks-ca" pod "54a3b34a-572b-4c88-b459-14d860ed649b" (UID: "54a3b34a-572b-4c88-b459-14d860ed649b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:24.152493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.151718    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b50207f-830b-4526-93b0-001008700eda-kube-api-access-rqzd2" pod "7b50207f-830b-4526-93b0-001008700eda" (UID: "7b50207f-830b-4526-93b0-001008700eda"). InnerVolumeSpecName "kube-api-access-rqzd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:24.152493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.151768    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a3b34a-572b-4c88-b459-14d860ed649b-kube-api-access-q9n4w" pod "54a3b34a-572b-4c88-b459-14d860ed649b" (UID: "54a3b34a-572b-4c88-b459-14d860ed649b"). InnerVolumeSpecName "kube-api-access-q9n4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:24.251009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.250966    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:24.251009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.251011    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b50207f-830b-4526-93b0-001008700eda-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:24.251294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.251030    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a3b34a-572b-4c88-b459-14d860ed649b-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:24.251294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.251045    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9n4w\" (UniqueName: \"kubernetes.io/projected/54a3b34a-572b-4c88-b459-14d860ed649b-kube-api-access-q9n4w\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:24.251294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.251061    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzd2\" (UniqueName: \"kubernetes.io/projected/7b50207f-830b-4526-93b0-001008700eda-kube-api-access-rqzd2\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:24.251294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.251075    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b50207f-830b-4526-93b0-001008700eda-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:24.919914 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.919885    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"
Oct 30 01:32:24.920152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.919920    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq" event={"ID":"54a3b34a-572b-4c88-b459-14d860ed649b","Type":"ContainerDied","Data":"ef1953d5784049e9daafd669fb42c2534b588b678fce715091d7dee58eea65fa"}
Oct 30 01:32:24.920152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.919970    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1953d5784049e9daafd669fb42c2534b588b678fce715091d7dee58eea65fa"
Oct 30 01:32:24.921766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.921745    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"
Oct 30 01:32:24.921936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.921788    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb" event={"ID":"7b50207f-830b-4526-93b0-001008700eda","Type":"ContainerDied","Data":"596d5dcc5c82b9883cc1a0350c83649731b0ca02e97ea7a536fa160a32dd3403"}
Oct 30 01:32:24.921936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:24.921854    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596d5dcc5c82b9883cc1a0350c83649731b0ca02e97ea7a536fa160a32dd3403"
Oct 30 01:32:25.533056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:25.533017    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" podUID="4e2470c5-091d-4c27-86b4-f096085566ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:32:27.703104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:27.703060    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main" containerID="cri-o://882abdf0000f6914c7c591e5a9142bfcfe75b3163002c5f166653890c0019517" gracePeriod=45
Oct 30 01:32:27.708911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:27.708878    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main" containerID="cri-o://49e6d4f359d1d91023efd765e156e1e05a9a22128fa2e2b6faa412fbfe88e3ac" gracePeriod=45
Oct 30 01:32:27.947412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:27.947375    5104 generic.go:356] "Generic (PLEG): container finished" podID="d5188ca8-b09c-476e-97e7-329244af24f3" containerID="882abdf0000f6914c7c591e5a9142bfcfe75b3163002c5f166653890c0019517" exitCode=0
Oct 30 01:32:27.947637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:27.947616    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" event={"ID":"d5188ca8-b09c-476e-97e7-329244af24f3","Type":"ContainerDied","Data":"882abdf0000f6914c7c591e5a9142bfcfe75b3163002c5f166653890c0019517"}
Oct 30 01:32:27.950160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:27.950130    5104 generic.go:356] "Generic (PLEG): container finished" podID="70f21280-ac89-4809-9826-26ad3e76020b" containerID="49e6d4f359d1d91023efd765e156e1e05a9a22128fa2e2b6faa412fbfe88e3ac" exitCode=0
Oct 30 01:32:27.950298 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:27.950232    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" event={"ID":"70f21280-ac89-4809-9826-26ad3e76020b","Type":"ContainerDied","Data":"49e6d4f359d1d91023efd765e156e1e05a9a22128fa2e2b6faa412fbfe88e3ac"}
Oct 30 01:32:27.982707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:27.982677    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:32:28.018696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.018655    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:32:28.091640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091483    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-dshm\") pod \"70f21280-ac89-4809-9826-26ad3e76020b\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") "
Oct 30 01:32:28.091640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091565    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-tmp-dir\") pod \"d5188ca8-b09c-476e-97e7-329244af24f3\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") "
Oct 30 01:32:28.091640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091642    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5188ca8-b09c-476e-97e7-329244af24f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5188ca8-b09c-476e-97e7-329244af24f3-tls-certs\") pod \"d5188ca8-b09c-476e-97e7-329244af24f3\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") "
Oct 30 01:32:28.091972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091682    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5188ca8-b09c-476e-97e7-329244af24f3-kube-api-access-svz9x\" (UniqueName: \"kubernetes.io/projected/d5188ca8-b09c-476e-97e7-329244af24f3-kube-api-access-svz9x\") pod \"d5188ca8-b09c-476e-97e7-329244af24f3\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") "
Oct 30 01:32:28.091972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091738    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-model-cache\") pod \"70f21280-ac89-4809-9826-26ad3e76020b\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") "
Oct 30 01:32:28.091972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091766    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-model-cache\") pod \"d5188ca8-b09c-476e-97e7-329244af24f3\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") "
Oct 30 01:32:28.091972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091794    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-home\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-home\") pod \"70f21280-ac89-4809-9826-26ad3e76020b\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") "
Oct 30 01:32:28.091972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091889    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-home\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-home\") pod \"d5188ca8-b09c-476e-97e7-329244af24f3\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") "
Oct 30 01:32:28.091972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091916    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-tmp-dir\") pod \"70f21280-ac89-4809-9826-26ad3e76020b\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") "
Oct 30 01:32:28.091972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091957    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f21280-ac89-4809-9826-26ad3e76020b-tls-certs\" (UniqueName: \"kubernetes.io/secret/70f21280-ac89-4809-9826-26ad3e76020b-tls-certs\") pod \"70f21280-ac89-4809-9826-26ad3e76020b\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") "
Oct 30 01:32:28.092321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.091992    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-dshm\") pod \"d5188ca8-b09c-476e-97e7-329244af24f3\" (UID: \"d5188ca8-b09c-476e-97e7-329244af24f3\") "
Oct 30 01:32:28.092321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092022    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f21280-ac89-4809-9826-26ad3e76020b-kube-api-access-tr56r\" (UniqueName: \"kubernetes.io/projected/70f21280-ac89-4809-9826-26ad3e76020b-kube-api-access-tr56r\") pod \"70f21280-ac89-4809-9826-26ad3e76020b\" (UID: \"70f21280-ac89-4809-9826-26ad3e76020b\") "
Oct 30 01:32:28.092321 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092100    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-model-cache" pod "70f21280-ac89-4809-9826-26ad3e76020b" (UID: "70f21280-ac89-4809-9826-26ad3e76020b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.092495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092334    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-home" pod "d5188ca8-b09c-476e-97e7-329244af24f3" (UID: "d5188ca8-b09c-476e-97e7-329244af24f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.092563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092528    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.092563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092536    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-model-cache" pod "d5188ca8-b09c-476e-97e7-329244af24f3" (UID: "d5188ca8-b09c-476e-97e7-329244af24f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.092563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092556    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.092747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092718    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-home" pod "70f21280-ac89-4809-9826-26ad3e76020b" (UID: "70f21280-ac89-4809-9826-26ad3e76020b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.092923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.092903    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-tmp-dir" pod "70f21280-ac89-4809-9826-26ad3e76020b" (UID: "70f21280-ac89-4809-9826-26ad3e76020b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.093803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.093772    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-tmp-dir" pod "d5188ca8-b09c-476e-97e7-329244af24f3" (UID: "d5188ca8-b09c-476e-97e7-329244af24f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.096664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.096613    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f21280-ac89-4809-9826-26ad3e76020b-kube-api-access-tr56r" pod "70f21280-ac89-4809-9826-26ad3e76020b" (UID: "70f21280-ac89-4809-9826-26ad3e76020b"). InnerVolumeSpecName "kube-api-access-tr56r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:28.096664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.096623    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f21280-ac89-4809-9826-26ad3e76020b-tls-certs" pod "70f21280-ac89-4809-9826-26ad3e76020b" (UID: "70f21280-ac89-4809-9826-26ad3e76020b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:28.097112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.097084    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-dshm" pod "70f21280-ac89-4809-9826-26ad3e76020b" (UID: "70f21280-ac89-4809-9826-26ad3e76020b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.097302 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.097265    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5188ca8-b09c-476e-97e7-329244af24f3-tls-certs" pod "d5188ca8-b09c-476e-97e7-329244af24f3" (UID: "d5188ca8-b09c-476e-97e7-329244af24f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:28.097973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.097945    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-dshm" pod "d5188ca8-b09c-476e-97e7-329244af24f3" (UID: "d5188ca8-b09c-476e-97e7-329244af24f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:28.098313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.098290    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5188ca8-b09c-476e-97e7-329244af24f3-kube-api-access-svz9x" pod "d5188ca8-b09c-476e-97e7-329244af24f3" (UID: "d5188ca8-b09c-476e-97e7-329244af24f3"). InnerVolumeSpecName "kube-api-access-svz9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:28.194138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194096    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5188ca8-b09c-476e-97e7-329244af24f3-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194131    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svz9x\" (UniqueName: \"kubernetes.io/projected/d5188ca8-b09c-476e-97e7-329244af24f3-kube-api-access-svz9x\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194143    5104 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-model-cache\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194152    5104 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-home\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194182    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194190    5104 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f21280-ac89-4809-9826-26ad3e76020b-tls-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194199    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194206    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr56r\" (UniqueName: \"kubernetes.io/projected/70f21280-ac89-4809-9826-26ad3e76020b-kube-api-access-tr56r\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194214    5104 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f21280-ac89-4809-9826-26ad3e76020b-dshm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.194577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.194223    5104 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5188ca8-b09c-476e-97e7-329244af24f3-tmp-dir\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:28.959539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.959509    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"
Oct 30 01:32:28.959539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.959518    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh" event={"ID":"70f21280-ac89-4809-9826-26ad3e76020b","Type":"ContainerDied","Data":"8eb274adfdf258ad9a6a0e74e5dc9f3368fa4702544ec2421aa1318fee912ea0"}
Oct 30 01:32:28.960122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.959572    5104 scope.go:122] "RemoveContainer" containerID="49e6d4f359d1d91023efd765e156e1e05a9a22128fa2e2b6faa412fbfe88e3ac"
Oct 30 01:32:28.961685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.961365    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8" event={"ID":"d5188ca8-b09c-476e-97e7-329244af24f3","Type":"ContainerDied","Data":"5de2df926fa4ec2abc42d761b5cb6a77bb81f720fc1db26551a30227ec9faeab"}
Oct 30 01:32:28.961685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.961370    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"
Oct 30 01:32:28.970551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:28.970515    5104 scope.go:122] "RemoveContainer" containerID="882abdf0000f6914c7c591e5a9142bfcfe75b3163002c5f166653890c0019517"
Oct 30 01:32:29.008655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:29.008617    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"]
Oct 30 01:32:29.019514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:29.019483    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-w6-c4847c-kserve-6f4cbf6fb6-fmwz8"]
Oct 30 01:32:29.041264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:29.041230    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"]
Oct 30 01:32:29.045489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:29.045435    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/premium-w6-c4847c-kserve-64f7d94747-74fbh"]
Oct 30 01:32:30.576288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:30.576252    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f21280-ac89-4809-9826-26ad3e76020b" path="/var/lib/kubelet/pods/70f21280-ac89-4809-9826-26ad3e76020b/volumes"
Oct 30 01:32:30.576824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:30.576707    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" path="/var/lib/kubelet/pods/d5188ca8-b09c-476e-97e7-329244af24f3/volumes"
Oct 30 01:32:33.824640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:33.824603    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:32:35.532236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:35.532195    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" podUID="4e2470c5-091d-4c27-86b4-f096085566ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:32:36.487219 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:36.487185    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"]
Oct 30 01:32:36.501207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:36.501173    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c4847c-7n8q8"]
Oct 30 01:32:36.576586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:36.576548    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eab6cbc-6c03-4c00-9a89-97386b957728" path="/var/lib/kubelet/pods/3eab6cbc-6c03-4c00-9a89-97386b957728/volumes"
Oct 30 01:32:39.104835 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:39.104790    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:32:39.701637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:39.701597    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:32:39.702127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:39.702101    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:32:39.709903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:39.709881    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:32:39.710063 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:39.709881    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:32:42.946941 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:42.946904    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"]
Oct 30 01:32:42.947493 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:42.947185    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" containerID="cri-o://a46dd6e396dec1a7c0b99391dcebd4e805f64f1c8b58ed461890e919494a0a56" gracePeriod=30
Oct 30 01:32:43.581976 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:43.581902    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 01:32:43.656633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:43.656584    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2470c5-091d-4c27-86b4-f096085566ca-kube-api-access-kdnjc\" (UniqueName: \"kubernetes.io/projected/4e2470c5-091d-4c27-86b4-f096085566ca-kube-api-access-kdnjc\") pod \"4e2470c5-091d-4c27-86b4-f096085566ca\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") "
Oct 30 01:32:43.656824 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:43.656654    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2470c5-091d-4c27-86b4-f096085566ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2470c5-091d-4c27-86b4-f096085566ca-plugins-config-volume\") pod \"4e2470c5-091d-4c27-86b4-f096085566ca\" (UID: \"4e2470c5-091d-4c27-86b4-f096085566ca\") "
Oct 30 01:32:43.657105 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:43.657076    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2470c5-091d-4c27-86b4-f096085566ca-plugins-config-volume" pod "4e2470c5-091d-4c27-86b4-f096085566ca" (UID: "4e2470c5-091d-4c27-86b4-f096085566ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:43.659080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:43.659050    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2470c5-091d-4c27-86b4-f096085566ca-kube-api-access-kdnjc" pod "4e2470c5-091d-4c27-86b4-f096085566ca" (UID: "4e2470c5-091d-4c27-86b4-f096085566ca"). InnerVolumeSpecName "kube-api-access-kdnjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:43.758308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:43.758262    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdnjc\" (UniqueName: \"kubernetes.io/projected/4e2470c5-091d-4c27-86b4-f096085566ca-kube-api-access-kdnjc\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:43.758308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:43.758302    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2470c5-091d-4c27-86b4-f096085566ca-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:44.074595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.074554    5104 generic.go:356] "Generic (PLEG): container finished" podID="4e2470c5-091d-4c27-86b4-f096085566ca" containerID="924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632" exitCode=0
Oct 30 01:32:44.075030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.074624    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"
Oct 30 01:32:44.075030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.074612    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" event={"ID":"4e2470c5-091d-4c27-86b4-f096085566ca","Type":"ContainerDied","Data":"924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632"}
Oct 30 01:32:44.075030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.074774    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r" event={"ID":"4e2470c5-091d-4c27-86b4-f096085566ca","Type":"ContainerDied","Data":"7c06e86916bf4473f50af603fe9c1a4787a12dfa8ae2ae950af9a20e58ffa261"}
Oct 30 01:32:44.075030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.074808    5104 scope.go:122] "RemoveContainer" containerID="924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632"
Oct 30 01:32:44.087609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.087591    5104 scope.go:122] "RemoveContainer" containerID="924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632"
Oct 30 01:32:44.087958 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:32:44.087936    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632\": container with ID starting with 924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632 not found: ID does not exist" containerID="924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632"
Oct 30 01:32:44.088056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.087968    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632"} err="failed to get container status \"924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632\": rpc error: code = NotFound desc = could not find container \"924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632\": container with ID starting with 924ff78c113d8a2747850c4a81417afaf0d47f3a10ffc75664541a89dced2632 not found: ID does not exist"
Oct 30 01:32:44.125790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.125748    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"]
Oct 30 01:32:44.136833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.136798    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c4847c-65c84d5d4c-cr22r"]
Oct 30 01:32:44.366022 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.365930    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:32:44.576679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.576623    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2470c5-091d-4c27-86b4-f096085566ca" path="/var/lib/kubelet/pods/4e2470c5-091d-4c27-86b4-f096085566ca/volumes"
Oct 30 01:32:44.713843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.713800    5104 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:32:44.714064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:44.713880    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:32:48.120994 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.120953    5104 generic.go:356] "Generic (PLEG): container finished" podID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerID="a46dd6e396dec1a7c0b99391dcebd4e805f64f1c8b58ed461890e919494a0a56" exitCode=0
Oct 30 01:32:48.121699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.121171    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" event={"ID":"d667e298-1030-4b99-999f-85b8f33bd4c9","Type":"ContainerDied","Data":"a46dd6e396dec1a7c0b99391dcebd4e805f64f1c8b58ed461890e919494a0a56"}
Oct 30 01:32:48.226264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.226185    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 01:32:48.401236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401202    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-socket\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401244    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-credential-socket\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401265    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-envoy\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401334    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-data\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401368    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-token\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-token\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401388    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-podinfo\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401407    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-certs\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401447    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-kube-api-access-vfjsr\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-kube-api-access-vfjsr\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401511    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d667e298-1030-4b99-999f-85b8f33bd4c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d667e298-1030-4b99-999f-85b8f33bd4c9-istiod-ca-cert\") pod \"d667e298-1030-4b99-999f-85b8f33bd4c9\" (UID: \"d667e298-1030-4b99-999f-85b8f33bd4c9\") "
Oct 30 01:32:48.401996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401550    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-credential-socket" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:48.401996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401614    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-socket" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:48.401996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401804    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-certs" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:48.401996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401915    5104 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.401996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401937    5104 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-workload-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.402292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401937    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d667e298-1030-4b99-999f-85b8f33bd4c9-istiod-ca-cert" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:48.402292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.402015    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-data" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:48.402292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.401955    5104 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-credential-socket\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.403865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.403839    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-podinfo" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:32:48.404194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.404169    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-token" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:48.404417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.404399    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-kube-api-access-vfjsr" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "kube-api-access-vfjsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:48.405605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.405577    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-envoy" pod "d667e298-1030-4b99-999f-85b8f33bd4c9" (UID: "d667e298-1030-4b99-999f-85b8f33bd4c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:48.503173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.503077    5104 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-data\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.503173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.503112    5104 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-token\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.503173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.503126    5104 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-podinfo\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.503173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.503137    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjsr\" (UniqueName: \"kubernetes.io/projected/d667e298-1030-4b99-999f-85b8f33bd4c9-kube-api-access-vfjsr\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.503173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.503150    5104 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d667e298-1030-4b99-999f-85b8f33bd4c9-istiod-ca-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.503173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.503163    5104 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d667e298-1030-4b99-999f-85b8f33bd4c9-istio-envoy\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:32:48.965236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.965187    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59cfcff4f-4mm7r"]
Oct 30 01:32:48.965998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.965943    5104 cpu_manager.go:414] "containerMap: removing container" podUID="54a3b34a-572b-4c88-b459-14d860ed649b" containerName="revoke-keys"
Oct 30 01:32:48.965998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.965974    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a3b34a-572b-4c88-b459-14d860ed649b" containerName="revoke-keys"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966005    5104 cpu_manager.go:414] "containerMap: removing container" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966015    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966046    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4e2470c5-091d-4c27-86b4-f096085566ca" containerName="payload-pre-processing"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966075    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2470c5-091d-4c27-86b4-f096085566ca" containerName="payload-pre-processing"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966089    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966097    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966114    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7b50207f-830b-4526-93b0-001008700eda" containerName="revoke-keys"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966123    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b50207f-830b-4526-93b0-001008700eda" containerName="revoke-keys"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966139    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy"
Oct 30 01:32:48.966218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966147    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy"
Oct 30 01:32:48.966840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966252    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a3b34a-572b-4c88-b459-14d860ed649b" containerName="revoke-keys"
Oct 30 01:32:48.966840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966267    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f21280-ac89-4809-9826-26ad3e76020b" containerName="main"
Oct 30 01:32:48.966840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966283    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2470c5-091d-4c27-86b4-f096085566ca" containerName="payload-pre-processing"
Oct 30 01:32:48.966840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966292    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5188ca8-b09c-476e-97e7-329244af24f3" containerName="main"
Oct 30 01:32:48.966840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966304    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b50207f-830b-4526-93b0-001008700eda" containerName="revoke-keys"
Oct 30 01:32:48.966840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.966315    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" containerName="istio-proxy"
Oct 30 01:32:48.970913 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.970889    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:48.994553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:48.994063    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-4mm7r"]
Oct 30 01:32:49.004053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.002073    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 01:32:49.004053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.003284    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 01:32:49.004053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.003390    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 01:32:49.004053 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.003514    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fmbv5\""
Oct 30 01:32:49.107886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.107838    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwgm\" (UniqueName: \"kubernetes.io/projected/60713e94-ac6e-4a54-8240-f240e1959436-kube-api-access-2lwgm\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.108086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.107918    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-metrics-certs\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.108142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.108107    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-webhook-cert\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.134443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.134398    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd" event={"ID":"d667e298-1030-4b99-999f-85b8f33bd4c9","Type":"ContainerDied","Data":"9dbdd73e863f7d29347b0e16be749df42c3b4a2f5be7d1b2cc26630bd600d9cf"}
Oct 30 01:32:49.134443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.134426    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"
Oct 30 01:32:49.135050 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.134503    5104 scope.go:122] "RemoveContainer" containerID="a46dd6e396dec1a7c0b99391dcebd4e805f64f1c8b58ed461890e919494a0a56"
Oct 30 01:32:49.185855 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.185818    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"]
Oct 30 01:32:49.189569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.189538    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-958pd"]
Oct 30 01:32:49.208776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.208738    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-webhook-cert\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.208974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.208827    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwgm\" (UniqueName: \"kubernetes.io/projected/60713e94-ac6e-4a54-8240-f240e1959436-kube-api-access-2lwgm\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.208974 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.208897    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-metrics-certs\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.211412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.211386    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-webhook-cert\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.211846 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.211826    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-metrics-certs\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.228950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.228885    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwgm\" (UniqueName: \"kubernetes.io/projected/60713e94-ac6e-4a54-8240-f240e1959436-kube-api-access-2lwgm\") pod \"maas-controller-59cfcff4f-4mm7r\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") " pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.304672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.304631    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:49.615823 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:49.615723    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-4mm7r"]
Oct 30 01:32:49.620179 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:32:49.620148    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60713e94_ac6e_4a54_8240_f240e1959436.slice/crio-7fada3bf2339e44d7ed1f423f7cd19cedd35d46ef3c1e899cafd221367e3f568 WatchSource:0}: Error finding container 7fada3bf2339e44d7ed1f423f7cd19cedd35d46ef3c1e899cafd221367e3f568: Status 404 returned error can't find the container with id 7fada3bf2339e44d7ed1f423f7cd19cedd35d46ef3c1e899cafd221367e3f568
Oct 30 01:32:50.149624 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:50.149582    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-4mm7r" event={"ID":"60713e94-ac6e-4a54-8240-f240e1959436","Type":"ContainerStarted","Data":"7fada3bf2339e44d7ed1f423f7cd19cedd35d46ef3c1e899cafd221367e3f568"}
Oct 30 01:32:50.576975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:50.576938    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d667e298-1030-4b99-999f-85b8f33bd4c9" path="/var/lib/kubelet/pods/d667e298-1030-4b99-999f-85b8f33bd4c9/volumes"
Oct 30 01:32:52.173025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:52.172982    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-4mm7r" event={"ID":"60713e94-ac6e-4a54-8240-f240e1959436","Type":"ContainerStarted","Data":"9f26bef57bfd15f9ff282b140c660b61ef6c3d25bd0c011deff8cb58c0b4786b"}
Oct 30 01:32:52.173540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:52.173049    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:32:52.210860 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:52.210806    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59cfcff4f-4mm7r" podStartSLOduration=2.560142904 podStartE2EDuration="4.210791537s" totalImagesPullingTime="1.650648633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:52.210064698 +0000 UTC m=+3013.757755195" watchObservedRunningTime="2026-10-30 01:32:52.210791537 +0000 UTC m=+3013.758482033"
Oct 30 01:32:55.187527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:55.187495    5104 scope.go:122] "RemoveContainer" containerID="9d1419047bfc583340709b24947da7cd535308a3ba9b30acddb9efca29f0f8ea"
Oct 30 01:32:55.197900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:32:55.197873    5104 scope.go:122] "RemoveContainer" containerID="838695f576db6cef86406bdd072fc9ecc02ba542c59f82574c5413169e577c13"
Oct 30 01:33:03.182843 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:03.182810    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:33:09.106179 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:33:09.106138    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:33:10.598870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.598831    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"]
Oct 30 01:33:10.602593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.602571    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.620414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.620385    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:33:10.620592 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.620528    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:33:10.632272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.632219    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:33:10.632492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.632272    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:33:10.634920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.634886    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"]
Oct 30 01:33:10.718731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.718694    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.718905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.718765    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcxk\" (UniqueName: \"kubernetes.io/projected/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-kube-api-access-dkcxk\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.718905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.718838    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.820106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.820061    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.820316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.820160    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcxk\" (UniqueName: \"kubernetes.io/projected/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-kube-api-access-dkcxk\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.820316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.820190    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.820959 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.820928    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.822764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.822739    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.839932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.839896    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcxk\" (UniqueName: \"kubernetes.io/projected/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-kube-api-access-dkcxk\") pod \"maas-api-revoke-sub-85237e45-2nq9t\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:10.916726 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:10.916633    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:11.212526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:11.212492    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"]
Oct 30 01:33:11.213941 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:33:11.213912    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60261e55_00cb_4c5e_b85c_bb0a6fdfdd0d.slice/crio-5e707cba342fcd1060f0b49e54b47513a4b851cb019bea33494061cf196bf81a WatchSource:0}: Error finding container 5e707cba342fcd1060f0b49e54b47513a4b851cb019bea33494061cf196bf81a: Status 404 returned error can't find the container with id 5e707cba342fcd1060f0b49e54b47513a4b851cb019bea33494061cf196bf81a
Oct 30 01:33:11.297380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:11.297343    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t" event={"ID":"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d","Type":"ContainerStarted","Data":"5e707cba342fcd1060f0b49e54b47513a4b851cb019bea33494061cf196bf81a"}
Oct 30 01:33:12.308188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:12.308149    5104 generic.go:356] "Generic (PLEG): container finished" podID="60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" containerID="b0db68b56a05c4d507a9e809cc89643ea96c2563552f053b2bb6108f9bac9e9b" exitCode=0
Oct 30 01:33:12.308604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:12.308234    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t" event={"ID":"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d","Type":"ContainerDied","Data":"b0db68b56a05c4d507a9e809cc89643ea96c2563552f053b2bb6108f9bac9e9b"}
Oct 30 01:33:13.467799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.467773    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:13.547293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.547255    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-kube-api-access-dkcxk\" (UniqueName: \"kubernetes.io/projected/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-kube-api-access-dkcxk\") pod \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") "
Oct 30 01:33:13.547521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.547337    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-xks-ca\") pod \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") "
Oct 30 01:33:13.547521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.547368    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-openshift-ca\") pod \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\" (UID: \"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d\") "
Oct 30 01:33:13.547978 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.547937    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-openshift-ca" pod "60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" (UID: "60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:33:13.549596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.549571    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-xks-ca" pod "60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" (UID: "60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:33:13.549704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.549646    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-kube-api-access-dkcxk" pod "60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" (UID: "60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d"). InnerVolumeSpecName "kube-api-access-dkcxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:33:13.606017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.605931    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"]
Oct 30 01:33:13.606352 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.606339    5104 cpu_manager.go:414] "containerMap: removing container" podUID="60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" containerName="revoke-keys"
Oct 30 01:33:13.606402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.606353    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" containerName="revoke-keys"
Oct 30 01:33:13.606466 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.606448    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" containerName="revoke-keys"
Oct 30 01:33:13.609783 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.609758    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.637014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.636985    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"]
Oct 30 01:33:13.648389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.648359    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95dpr\" (UniqueName: \"kubernetes.io/projected/02175766-c6b6-45ca-86e2-fc737cc0cfc3-kube-api-access-95dpr\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.648565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.648409    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.648725 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.648653    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.648992 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.648957    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.649147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649002    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.649147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649074    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.649261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649183    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.649261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649235    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.649572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649343    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.649572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649517    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkcxk\" (UniqueName: \"kubernetes.io/projected/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-kube-api-access-dkcxk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:33:13.649572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649547    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:33:13.649572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.649563    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:33:13.750612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750571    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dpr\" (UniqueName: \"kubernetes.io/projected/02175766-c6b6-45ca-86e2-fc737cc0cfc3-kube-api-access-95dpr\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750620    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750649    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750684    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750713    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750737    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750789    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750816    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.750903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.750863    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.751268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.751109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.751268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.751154    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.751268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.751236    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.751437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.751404    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.751565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.751541    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.753240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.753216    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.753505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.753482    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.770341 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.770309    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02175766-c6b6-45ca-86e2-fc737cc0cfc3-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.770495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.770478    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dpr\" (UniqueName: \"kubernetes.io/projected/02175766-c6b6-45ca-86e2-fc737cc0cfc3-kube-api-access-95dpr\") pod \"maas-default-gateway-openshift-default-67bf986f99-92m4m\" (UID: \"02175766-c6b6-45ca-86e2-fc737cc0cfc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:13.922647 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:13.922614    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:14.236293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.236215    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"]
Oct 30 01:33:14.238718 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:33:14.238684    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02175766_c6b6_45ca_86e2_fc737cc0cfc3.slice/crio-5f744f511fb594978d01c179c301c8f6740d8186f9b023d143df2df60da361e8 WatchSource:0}: Error finding container 5f744f511fb594978d01c179c301c8f6740d8186f9b023d143df2df60da361e8: Status 404 returned error can't find the container with id 5f744f511fb594978d01c179c301c8f6740d8186f9b023d143df2df60da361e8
Oct 30 01:33:14.241014 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.240981    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:33:14.241086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.241049    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:33:14.241086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.241079    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:33:14.322797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.322764    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"
Oct 30 01:33:14.323002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.322767    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t" event={"ID":"60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d","Type":"ContainerDied","Data":"5e707cba342fcd1060f0b49e54b47513a4b851cb019bea33494061cf196bf81a"}
Oct 30 01:33:14.323002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.322904    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e707cba342fcd1060f0b49e54b47513a4b851cb019bea33494061cf196bf81a"
Oct 30 01:33:14.324892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.324858    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" event={"ID":"02175766-c6b6-45ca-86e2-fc737cc0cfc3","Type":"ContainerStarted","Data":"f368c99086874aa84f7b692e76183beded2f527ecb223812f625506f7454261b"}
Oct 30 01:33:14.325045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.324913    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" event={"ID":"02175766-c6b6-45ca-86e2-fc737cc0cfc3","Type":"ContainerStarted","Data":"5f744f511fb594978d01c179c301c8f6740d8186f9b023d143df2df60da361e8"}
Oct 30 01:33:14.367205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.367141    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podStartSLOduration=1.367121772 podStartE2EDuration="1.367121772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:14.364651833 +0000 UTC m=+3035.912342329" watchObservedRunningTime="2026-10-30 01:33:14.367121772 +0000 UTC m=+3035.914812270"
Oct 30 01:33:14.923156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:14.923119    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:15.027437 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:15.027399    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"]
Oct 30 01:33:15.038328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:15.038292    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"]
Oct 30 01:33:15.040656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:15.040629    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c7da5-7jv5j"]
Oct 30 01:33:15.042737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:15.042713    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5159e6-289pm"]
Oct 30 01:33:15.924901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:15.924866    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-92m4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body=
Oct 30 01:33:15.925317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:15.924924    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podUID="02175766-c6b6-45ca-86e2-fc737cc0cfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused"
Oct 30 01:33:16.599427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:16.599383    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04824d47-54d7-40f4-a83c-f53fac403ef4" path="/var/lib/kubelet/pods/04824d47-54d7-40f4-a83c-f53fac403ef4/volumes"
Oct 30 01:33:16.600030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:16.600007    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ad2cf2-974b-4f97-b873-24901d81153a" path="/var/lib/kubelet/pods/97ad2cf2-974b-4f97-b873-24901d81153a/volumes"
Oct 30 01:33:16.923256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:16.923224    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-92m4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body=
Oct 30 01:33:16.923434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:16.923278    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podUID="02175766-c6b6-45ca-86e2-fc737cc0cfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused"
Oct 30 01:33:17.923670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:17.923628    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-92m4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body=
Oct 30 01:33:17.924071 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:17.923697    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podUID="02175766-c6b6-45ca-86e2-fc737cc0cfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused"
Oct 30 01:33:18.923483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:18.923412    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-92m4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body=
Oct 30 01:33:18.923657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:18.923500    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podUID="02175766-c6b6-45ca-86e2-fc737cc0cfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused"
Oct 30 01:33:19.923597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:19.923551    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-92m4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body=
Oct 30 01:33:19.924103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:19.923616    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podUID="02175766-c6b6-45ca-86e2-fc737cc0cfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused"
Oct 30 01:33:20.923210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:20.923170    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-92m4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body=
Oct 30 01:33:20.923404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:20.923232    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podUID="02175766-c6b6-45ca-86e2-fc737cc0cfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused"
Oct 30 01:33:22.228488 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.228429    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-4mm7r"]
Oct 30 01:33:22.229060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.228850    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-59cfcff4f-4mm7r" podUID="60713e94-ac6e-4a54-8240-f240e1959436" containerName="manager" containerID="cri-o://9f26bef57bfd15f9ff282b140c660b61ef6c3d25bd0c011deff8cb58c0b4786b" gracePeriod=30
Oct 30 01:33:22.383062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.383027    5104 generic.go:356] "Generic (PLEG): container finished" podID="60713e94-ac6e-4a54-8240-f240e1959436" containerID="9f26bef57bfd15f9ff282b140c660b61ef6c3d25bd0c011deff8cb58c0b4786b" exitCode=0
Oct 30 01:33:22.383272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.383236    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-4mm7r" event={"ID":"60713e94-ac6e-4a54-8240-f240e1959436","Type":"ContainerDied","Data":"9f26bef57bfd15f9ff282b140c660b61ef6c3d25bd0c011deff8cb58c0b4786b"}
Oct 30 01:33:22.505316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.505279    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:33:22.662426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.662389    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-metrics-certs\") pod \"60713e94-ac6e-4a54-8240-f240e1959436\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") "
Oct 30 01:33:22.662426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.662431    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-webhook-cert\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-webhook-cert\") pod \"60713e94-ac6e-4a54-8240-f240e1959436\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") "
Oct 30 01:33:22.662676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.662530    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60713e94-ac6e-4a54-8240-f240e1959436-kube-api-access-2lwgm\" (UniqueName: \"kubernetes.io/projected/60713e94-ac6e-4a54-8240-f240e1959436-kube-api-access-2lwgm\") pod \"60713e94-ac6e-4a54-8240-f240e1959436\" (UID: \"60713e94-ac6e-4a54-8240-f240e1959436\") "
Oct 30 01:33:22.664713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.664656    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-metrics-certs" pod "60713e94-ac6e-4a54-8240-f240e1959436" (UID: "60713e94-ac6e-4a54-8240-f240e1959436"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:33:22.664713 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.664680    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-webhook-cert" pod "60713e94-ac6e-4a54-8240-f240e1959436" (UID: "60713e94-ac6e-4a54-8240-f240e1959436"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:33:22.665201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.665167    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60713e94-ac6e-4a54-8240-f240e1959436-kube-api-access-2lwgm" pod "60713e94-ac6e-4a54-8240-f240e1959436" (UID: "60713e94-ac6e-4a54-8240-f240e1959436"). InnerVolumeSpecName "kube-api-access-2lwgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:33:22.671476 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.671431    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:22.671983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.671964    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:22.672901 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.672876    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m"
Oct 30 01:33:22.764261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.764161    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lwgm\" (UniqueName: \"kubernetes.io/projected/60713e94-ac6e-4a54-8240-f240e1959436-kube-api-access-2lwgm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:33:22.764261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.764197    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:33:22.764261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:22.764210    5104 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60713e94-ac6e-4a54-8240-f240e1959436-webhook-cert\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:33:23.391981 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:23.391952    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-4mm7r"
Oct 30 01:33:23.392539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:23.391984    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-4mm7r" event={"ID":"60713e94-ac6e-4a54-8240-f240e1959436","Type":"ContainerDied","Data":"7fada3bf2339e44d7ed1f423f7cd19cedd35d46ef3c1e899cafd221367e3f568"}
Oct 30 01:33:23.392539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:23.392052    5104 scope.go:122] "RemoveContainer" containerID="9f26bef57bfd15f9ff282b140c660b61ef6c3d25bd0c011deff8cb58c0b4786b"
Oct 30 01:33:23.441830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:23.441790    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-4mm7r"]
Oct 30 01:33:23.453138 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:23.453099    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-4mm7r"]
Oct 30 01:33:24.576556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.576518    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60713e94-ac6e-4a54-8240-f240e1959436" path="/var/lib/kubelet/pods/60713e94-ac6e-4a54-8240-f240e1959436/volumes"
Oct 30 01:33:24.600889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.600857    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59cfcff4f-m8snp"]
Oct 30 01:33:24.601342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.601325    5104 cpu_manager.go:414] "containerMap: removing container" podUID="60713e94-ac6e-4a54-8240-f240e1959436" containerName="manager"
Oct 30 01:33:24.601342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.601342    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60713e94-ac6e-4a54-8240-f240e1959436" containerName="manager"
Oct 30 01:33:24.601527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.601427    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="60713e94-ac6e-4a54-8240-f240e1959436" containerName="manager"
Oct 30 01:33:24.607952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.607925    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.622008 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.621976    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 01:33:24.633498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.633444    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fmbv5\""
Oct 30 01:33:24.633668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.633560    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 01:33:24.633668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.633636    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 01:33:24.636002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.635977    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-m8snp"]
Oct 30 01:33:24.683179 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.683145    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4722ec-8353-403a-ab6b-864d3bc1b60d-metrics-certs\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.683399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.683276    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wlm\" (UniqueName: \"kubernetes.io/projected/dc4722ec-8353-403a-ab6b-864d3bc1b60d-kube-api-access-f8wlm\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.683399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.683328    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4722ec-8353-403a-ab6b-864d3bc1b60d-webhook-cert\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.784316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.784282    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4722ec-8353-403a-ab6b-864d3bc1b60d-metrics-certs\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.784586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.784359    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wlm\" (UniqueName: \"kubernetes.io/projected/dc4722ec-8353-403a-ab6b-864d3bc1b60d-kube-api-access-f8wlm\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.784586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.784385    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4722ec-8353-403a-ab6b-864d3bc1b60d-webhook-cert\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.787012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.786979    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4722ec-8353-403a-ab6b-864d3bc1b60d-metrics-certs\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.787012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.786996    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4722ec-8353-403a-ab6b-864d3bc1b60d-webhook-cert\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.803949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.803922    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wlm\" (UniqueName: \"kubernetes.io/projected/dc4722ec-8353-403a-ab6b-864d3bc1b60d-kube-api-access-f8wlm\") pod \"maas-controller-59cfcff4f-m8snp\" (UID: \"dc4722ec-8353-403a-ab6b-864d3bc1b60d\") " pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:24.922829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:24.922799    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:25.218812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:25.218775    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-m8snp"]
Oct 30 01:33:25.221185 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:33:25.221157    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4722ec_8353_403a_ab6b_864d3bc1b60d.slice/crio-00b5556aae29132728c9a611f5b8ba8e6179225d972f7f25ccbe112766971592 WatchSource:0}: Error finding container 00b5556aae29132728c9a611f5b8ba8e6179225d972f7f25ccbe112766971592: Status 404 returned error can't find the container with id 00b5556aae29132728c9a611f5b8ba8e6179225d972f7f25ccbe112766971592
Oct 30 01:33:25.222541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:25.222521    5104 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:33:25.409983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:25.409945    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-m8snp" event={"ID":"dc4722ec-8353-403a-ab6b-864d3bc1b60d","Type":"ContainerStarted","Data":"00b5556aae29132728c9a611f5b8ba8e6179225d972f7f25ccbe112766971592"}
Oct 30 01:33:26.418658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:26.418599    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-m8snp" event={"ID":"dc4722ec-8353-403a-ab6b-864d3bc1b60d","Type":"ContainerStarted","Data":"872897afbec1b507f6294553a542edcb8fc5897f79c88fb1f6b05a30fedc020c"}
Oct 30 01:33:26.419087 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:26.418682    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:26.460562 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:26.460507    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59cfcff4f-m8snp" podStartSLOduration=2.138905586 podStartE2EDuration="2.460486837s" totalImagesPullingTime="321.581251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:26.459539554 +0000 UTC m=+3048.007230050" watchObservedRunningTime="2026-10-30 01:33:26.460486837 +0000 UTC m=+3048.008177334"
Oct 30 01:33:37.427258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:37.427217    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59cfcff4f-m8snp"
Oct 30 01:33:39.103273 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:33:39.103234    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:33:53.244920 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:53.244878    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:33:55.285979 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:55.285936    5104 scope.go:122] "RemoveContainer" containerID="3c51d6c4f8517d5723699bca41b1b2df8e9f2e588a4acc7fefd26598e19ddddb"
Oct 30 01:33:55.303551 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:55.303517    5104 scope.go:122] "RemoveContainer" containerID="a112d3874a93c60573a240b2ec0fb9dc9c843c47dbf61cc1cc657f70a18c3925"
Oct 30 01:33:55.321303 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:55.321270    5104 scope.go:122] "RemoveContainer" containerID="9d78f75423346437967401e0edb140ed42b5fb3e2b91abb9952ca63a2b2a0e74"
Oct 30 01:33:55.337583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:33:55.337552    5104 scope.go:122] "RemoveContainer" containerID="249f9ac4dab7457a4d05b8afd196ab0a8218c8990a90f636c912807754c1150a"
Oct 30 01:34:00.179430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.179390    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"]
Oct 30 01:34:00.184982 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.184955    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.202529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.202469    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"]
Oct 30 01:34:00.211508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.211475    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:34:00.211760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.211491    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:34:00.211760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.211517    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:34:00.211988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.211568    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:34:00.356939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.356892    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59x9\" (UniqueName: \"kubernetes.io/projected/b128c1ba-58de-401e-a359-8099d2de926b-kube-api-access-g59x9\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.357158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.356959    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.357158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.357095    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.458259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.458146    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.458259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.458230    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59x9\" (UniqueName: \"kubernetes.io/projected/b128c1ba-58de-401e-a359-8099d2de926b-kube-api-access-g59x9\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.458610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.458291    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.458944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.458917    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.460776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.460751    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.478205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.478166    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59x9\" (UniqueName: \"kubernetes.io/projected/b128c1ba-58de-401e-a359-8099d2de926b-kube-api-access-g59x9\") pod \"maas-api-revoke-sub-c1e271a9-kpvhf\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.498074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.498039    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:00.798052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:00.798024    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"]
Oct 30 01:34:00.800419 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:34:00.800385    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb128c1ba_58de_401e_a359_8099d2de926b.slice/crio-547d941440a6cb9b255cf662c4440f6bc587b6fa6a60e0ad567d0ba6e80647ee WatchSource:0}: Error finding container 547d941440a6cb9b255cf662c4440f6bc587b6fa6a60e0ad567d0ba6e80647ee: Status 404 returned error can't find the container with id 547d941440a6cb9b255cf662c4440f6bc587b6fa6a60e0ad567d0ba6e80647ee
Oct 30 01:34:01.673717 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:01.673673    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf" event={"ID":"b128c1ba-58de-401e-a359-8099d2de926b","Type":"ContainerStarted","Data":"6734574f00f14ab06a3abd5887881478bdb214734fc97b7e8ded2918d5f8e8d0"}
Oct 30 01:34:01.674235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:01.673737    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf" event={"ID":"b128c1ba-58de-401e-a359-8099d2de926b","Type":"ContainerStarted","Data":"547d941440a6cb9b255cf662c4440f6bc587b6fa6a60e0ad567d0ba6e80647ee"}
Oct 30 01:34:01.712402 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:01.712346    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf" podStartSLOduration=1.712332529 podStartE2EDuration="1.712332529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:01.710988282 +0000 UTC m=+3083.258678778" watchObservedRunningTime="2026-10-30 01:34:01.712332529 +0000 UTC m=+3083.260023034"
Oct 30 01:34:02.682892 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:02.682853    5104 generic.go:356] "Generic (PLEG): container finished" podID="b128c1ba-58de-401e-a359-8099d2de926b" containerID="6734574f00f14ab06a3abd5887881478bdb214734fc97b7e8ded2918d5f8e8d0" exitCode=0
Oct 30 01:34:02.683382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:02.682939    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf" event={"ID":"b128c1ba-58de-401e-a359-8099d2de926b","Type":"ContainerDied","Data":"6734574f00f14ab06a3abd5887881478bdb214734fc97b7e8ded2918d5f8e8d0"}
Oct 30 01:34:03.839934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.839902    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:03.996385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.996280    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-xks-ca\") pod \"b128c1ba-58de-401e-a359-8099d2de926b\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") "
Oct 30 01:34:03.996385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.996331    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-openshift-ca\") pod \"b128c1ba-58de-401e-a359-8099d2de926b\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") "
Oct 30 01:34:03.996385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.996385    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b128c1ba-58de-401e-a359-8099d2de926b-kube-api-access-g59x9\" (UniqueName: \"kubernetes.io/projected/b128c1ba-58de-401e-a359-8099d2de926b-kube-api-access-g59x9\") pod \"b128c1ba-58de-401e-a359-8099d2de926b\" (UID: \"b128c1ba-58de-401e-a359-8099d2de926b\") "
Oct 30 01:34:03.996772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.996720    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-openshift-ca" pod "b128c1ba-58de-401e-a359-8099d2de926b" (UID: "b128c1ba-58de-401e-a359-8099d2de926b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:03.996858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.996838    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:03.998575 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.998550    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-xks-ca" pod "b128c1ba-58de-401e-a359-8099d2de926b" (UID: "b128c1ba-58de-401e-a359-8099d2de926b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:03.998704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:03.998627    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b128c1ba-58de-401e-a359-8099d2de926b-kube-api-access-g59x9" pod "b128c1ba-58de-401e-a359-8099d2de926b" (UID: "b128c1ba-58de-401e-a359-8099d2de926b"). InnerVolumeSpecName "kube-api-access-g59x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:04.097560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:04.097516    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b128c1ba-58de-401e-a359-8099d2de926b-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:04.097560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:04.097547    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g59x9\" (UniqueName: \"kubernetes.io/projected/b128c1ba-58de-401e-a359-8099d2de926b-kube-api-access-g59x9\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:04.697404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:04.697365    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"
Oct 30 01:34:04.697404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:04.697390    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf" event={"ID":"b128c1ba-58de-401e-a359-8099d2de926b","Type":"ContainerDied","Data":"547d941440a6cb9b255cf662c4440f6bc587b6fa6a60e0ad567d0ba6e80647ee"}
Oct 30 01:34:04.697655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:04.697476    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547d941440a6cb9b255cf662c4440f6bc587b6fa6a60e0ad567d0ba6e80647ee"
Oct 30 01:34:09.103512 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:34:09.103471    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:34:11.408937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:11.408896    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:34:16.007360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:16.007314    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:34:19.906776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.906672    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"]
Oct 30 01:34:19.907412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.907313    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b128c1ba-58de-401e-a359-8099d2de926b" containerName="revoke-keys"
Oct 30 01:34:19.907412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.907339    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b128c1ba-58de-401e-a359-8099d2de926b" containerName="revoke-keys"
Oct 30 01:34:19.907605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.907512    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b128c1ba-58de-401e-a359-8099d2de926b" containerName="revoke-keys"
Oct 30 01:34:19.911567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.911535    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:19.936708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.936675    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:34:19.936935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.936684    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:34:19.936935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.936734    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:34:19.936935 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.936740    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:34:19.939325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.939279    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"]
Oct 30 01:34:19.962181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.962140    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:19.962409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.962272    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:19.962513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:19.962434    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ss9c\" (UniqueName: \"kubernetes.io/projected/c8e02772-efd4-494e-a0f6-3de43112e62b-kube-api-access-4ss9c\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.062989 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.062946    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.063185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.063048    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ss9c\" (UniqueName: \"kubernetes.io/projected/c8e02772-efd4-494e-a0f6-3de43112e62b-kube-api-access-4ss9c\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.063185 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.063096    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.063875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.063852    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.065529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.065502    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.084064 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.084029    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ss9c\" (UniqueName: \"kubernetes.io/projected/c8e02772-efd4-494e-a0f6-3de43112e62b-kube-api-access-4ss9c\") pod \"maas-api-revoke-sub-2e1f5b5f-w456d\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.225757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.225712    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:20.521595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.521561    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"]
Oct 30 01:34:20.523589 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:34:20.523558    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e02772_efd4_494e_a0f6_3de43112e62b.slice/crio-9bd9f8e63c60d5d6af47b6f27d0bbd386cc546a16bfa84fb31148b2214150c63 WatchSource:0}: Error finding container 9bd9f8e63c60d5d6af47b6f27d0bbd386cc546a16bfa84fb31148b2214150c63: Status 404 returned error can't find the container with id 9bd9f8e63c60d5d6af47b6f27d0bbd386cc546a16bfa84fb31148b2214150c63
Oct 30 01:34:20.869930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.869884    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d" event={"ID":"c8e02772-efd4-494e-a0f6-3de43112e62b","Type":"ContainerStarted","Data":"cb573bafa7e14eef98df02388a87c723ff2de867bfb472a3511ca31096fff7bf"}
Oct 30 01:34:20.870134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.869945    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d" event={"ID":"c8e02772-efd4-494e-a0f6-3de43112e62b","Type":"ContainerStarted","Data":"9bd9f8e63c60d5d6af47b6f27d0bbd386cc546a16bfa84fb31148b2214150c63"}
Oct 30 01:34:20.906513 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:20.906425    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d" podStartSLOduration=1.9064103860000001 podStartE2EDuration="1.906410386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:20.906295178 +0000 UTC m=+3102.453985673" watchObservedRunningTime="2026-10-30 01:34:20.906410386 +0000 UTC m=+3102.454100882"
Oct 30 01:34:21.879447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:21.879409    5104 generic.go:356] "Generic (PLEG): container finished" podID="c8e02772-efd4-494e-a0f6-3de43112e62b" containerID="cb573bafa7e14eef98df02388a87c723ff2de867bfb472a3511ca31096fff7bf" exitCode=0
Oct 30 01:34:21.879985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:21.879500    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d" event={"ID":"c8e02772-efd4-494e-a0f6-3de43112e62b","Type":"ContainerDied","Data":"cb573bafa7e14eef98df02388a87c723ff2de867bfb472a3511ca31096fff7bf"}
Oct 30 01:34:23.034685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.034655    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:23.096939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.096900    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e02772-efd4-494e-a0f6-3de43112e62b-kube-api-access-4ss9c\" (UniqueName: \"kubernetes.io/projected/c8e02772-efd4-494e-a0f6-3de43112e62b-kube-api-access-4ss9c\") pod \"c8e02772-efd4-494e-a0f6-3de43112e62b\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") "
Oct 30 01:34:23.096939 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.096946    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-openshift-ca\") pod \"c8e02772-efd4-494e-a0f6-3de43112e62b\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") "
Oct 30 01:34:23.097184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.097030    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-xks-ca\") pod \"c8e02772-efd4-494e-a0f6-3de43112e62b\" (UID: \"c8e02772-efd4-494e-a0f6-3de43112e62b\") "
Oct 30 01:34:23.097393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.097366    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-openshift-ca" pod "c8e02772-efd4-494e-a0f6-3de43112e62b" (UID: "c8e02772-efd4-494e-a0f6-3de43112e62b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:23.099262 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.099223    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-xks-ca" pod "c8e02772-efd4-494e-a0f6-3de43112e62b" (UID: "c8e02772-efd4-494e-a0f6-3de43112e62b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:23.099411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.099323    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e02772-efd4-494e-a0f6-3de43112e62b-kube-api-access-4ss9c" pod "c8e02772-efd4-494e-a0f6-3de43112e62b" (UID: "c8e02772-efd4-494e-a0f6-3de43112e62b"). InnerVolumeSpecName "kube-api-access-4ss9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:23.198412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.198367    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:23.198412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.198406    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ss9c\" (UniqueName: \"kubernetes.io/projected/c8e02772-efd4-494e-a0f6-3de43112e62b-kube-api-access-4ss9c\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:23.198412 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.198417    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e02772-efd4-494e-a0f6-3de43112e62b-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:23.894596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.894558    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d" event={"ID":"c8e02772-efd4-494e-a0f6-3de43112e62b","Type":"ContainerDied","Data":"9bd9f8e63c60d5d6af47b6f27d0bbd386cc546a16bfa84fb31148b2214150c63"}
Oct 30 01:34:23.894596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.894586    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"
Oct 30 01:34:23.894878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:23.894612    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bd9f8e63c60d5d6af47b6f27d0bbd386cc546a16bfa84fb31148b2214150c63"
Oct 30 01:34:28.850320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.850282    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"]
Oct 30 01:34:28.850925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.850817    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c8e02772-efd4-494e-a0f6-3de43112e62b" containerName="revoke-keys"
Oct 30 01:34:28.850925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.850832    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e02772-efd4-494e-a0f6-3de43112e62b" containerName="revoke-keys"
Oct 30 01:34:28.851025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.850935    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e02772-efd4-494e-a0f6-3de43112e62b" containerName="revoke-keys"
Oct 30 01:34:28.854390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.854369    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.883268 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.883224    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"]
Oct 30 01:34:28.954445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954402    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954445 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954449    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954584    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954702 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954676    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954705    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954780 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954736    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954857 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954784    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954856    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc487\" (UniqueName: \"kubernetes.io/projected/042bc9a4-74e8-4187-ac04-6afbf483756d-kube-api-access-fc487\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:28.954898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:28.954890    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042bc9a4-74e8-4187-ac04-6afbf483756d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.055727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.055683    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.055727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.055731    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc487\" (UniqueName: \"kubernetes.io/projected/042bc9a4-74e8-4187-ac04-6afbf483756d-kube-api-access-fc487\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.055763    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042bc9a4-74e8-4187-ac04-6afbf483756d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.055802    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.055841    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.055910    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056086    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056137    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056191    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056269 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056187    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056304    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056331    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056569    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.056727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.056631    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042bc9a4-74e8-4187-ac04-6afbf483756d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.058276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.058252    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.058521 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.058501    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.074190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.074162    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042bc9a4-74e8-4187-ac04-6afbf483756d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.074409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.074389    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc487\" (UniqueName: \"kubernetes.io/projected/042bc9a4-74e8-4187-ac04-6afbf483756d-kube-api-access-fc487\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwnrr\" (UID: \"042bc9a4-74e8-4187-ac04-6afbf483756d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.169344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.169310    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:29.479305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.479274    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"]
Oct 30 01:34:29.481508 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:34:29.481479    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042bc9a4_74e8_4187_ac04_6afbf483756d.slice/crio-feef3a2a015a6279634f03fa0266a5c720324aba2bce91ae245f97e092469260 WatchSource:0}: Error finding container feef3a2a015a6279634f03fa0266a5c720324aba2bce91ae245f97e092469260: Status 404 returned error can't find the container with id feef3a2a015a6279634f03fa0266a5c720324aba2bce91ae245f97e092469260
Oct 30 01:34:29.483762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.483730    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:34:29.483851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.483795    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:34:29.483851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.483826    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:34:29.942937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.942893    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" event={"ID":"042bc9a4-74e8-4187-ac04-6afbf483756d","Type":"ContainerStarted","Data":"eef1f8160959d6adcdd6fa0cb371e91a37d5452805e1983da631d27f67b1e3c5"}
Oct 30 01:34:29.943447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.942980    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" event={"ID":"042bc9a4-74e8-4187-ac04-6afbf483756d","Type":"ContainerStarted","Data":"feef3a2a015a6279634f03fa0266a5c720324aba2bce91ae245f97e092469260"}
Oct 30 01:34:29.998998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:29.998928    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podStartSLOduration=1.998908382 podStartE2EDuration="1.998908382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:29.997901924 +0000 UTC m=+3111.545592422" watchObservedRunningTime="2026-10-30 01:34:29.998908382 +0000 UTC m=+3111.546598879"
Oct 30 01:34:30.170451 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:30.170408    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:30.171932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:30.171903    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:30.172002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:30.171960    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:31.170622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.170578    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:31.171142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.170646    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:31.199480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.199077    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"]
Oct 30 01:34:31.207085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.207049    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.213310 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.213233    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"]
Oct 30 01:34:31.221486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.221441    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:34:31.221664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.221592    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:34:31.229739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.229709    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:34:31.229906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.229770    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:34:31.241613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.241575    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:34:31.304186 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.304146    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.304408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.304214    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m74vp\" (UniqueName: \"kubernetes.io/projected/b72a9a3f-1065-40e4-a82d-189cdea16a39-kube-api-access-m74vp\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.304408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.304341    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.405590 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.405544    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.405790 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.405620    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m74vp\" (UniqueName: \"kubernetes.io/projected/b72a9a3f-1065-40e4-a82d-189cdea16a39-kube-api-access-m74vp\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.405850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.405796    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.406340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.406308    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.408699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.408674    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.424599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.424530    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m74vp\" (UniqueName: \"kubernetes.io/projected/b72a9a3f-1065-40e4-a82d-189cdea16a39-kube-api-access-m74vp\") pod \"maas-api-revoke-sub-ec843018-r5qwb\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.526390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.526342    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:31.870338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.870294    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"]
Oct 30 01:34:31.872572 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:34:31.872534    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72a9a3f_1065_40e4_a82d_189cdea16a39.slice/crio-5c082cb11d0441d6d51e3d9a20d2d91abe0d3480cba34ec7a04d77079e70643c WatchSource:0}: Error finding container 5c082cb11d0441d6d51e3d9a20d2d91abe0d3480cba34ec7a04d77079e70643c: Status 404 returned error can't find the container with id 5c082cb11d0441d6d51e3d9a20d2d91abe0d3480cba34ec7a04d77079e70643c
Oct 30 01:34:31.959001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:31.958961    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb" event={"ID":"b72a9a3f-1065-40e4-a82d-189cdea16a39","Type":"ContainerStarted","Data":"5c082cb11d0441d6d51e3d9a20d2d91abe0d3480cba34ec7a04d77079e70643c"}
Oct 30 01:34:32.170646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:32.170612    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:32.171040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:32.170672    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:32.966794 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:32.966757    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb" event={"ID":"b72a9a3f-1065-40e4-a82d-189cdea16a39","Type":"ContainerStarted","Data":"9299b2fc2faa2db12e9c0d2286b7658c01edeefa40531f6b4631a151633d6580"}
Oct 30 01:34:33.010829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:33.010742    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb" podStartSLOduration=2.010691043 podStartE2EDuration="2.010691043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:33.010003577 +0000 UTC m=+3114.557694072" watchObservedRunningTime="2026-10-30 01:34:33.010691043 +0000 UTC m=+3114.558381539"
Oct 30 01:34:33.170030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:33.169980    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:33.170251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:33.170043    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:33.983480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:33.982247    5104 generic.go:356] "Generic (PLEG): container finished" podID="b72a9a3f-1065-40e4-a82d-189cdea16a39" containerID="9299b2fc2faa2db12e9c0d2286b7658c01edeefa40531f6b4631a151633d6580" exitCode=0
Oct 30 01:34:33.983480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:33.982426    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb" event={"ID":"b72a9a3f-1065-40e4-a82d-189cdea16a39","Type":"ContainerDied","Data":"9299b2fc2faa2db12e9c0d2286b7658c01edeefa40531f6b4631a151633d6580"}
Oct 30 01:34:34.170270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:34.170233    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:34.170511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:34.170287    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:35.170037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.169988    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:35.170584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.170047    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:35.220760 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.220727    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:35.346555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.346431    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-xks-ca\") pod \"b72a9a3f-1065-40e4-a82d-189cdea16a39\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") "
Oct 30 01:34:35.346749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.346558    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72a9a3f-1065-40e4-a82d-189cdea16a39-kube-api-access-m74vp\" (UniqueName: \"kubernetes.io/projected/b72a9a3f-1065-40e4-a82d-189cdea16a39-kube-api-access-m74vp\") pod \"b72a9a3f-1065-40e4-a82d-189cdea16a39\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") "
Oct 30 01:34:35.346749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.346630    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-openshift-ca\") pod \"b72a9a3f-1065-40e4-a82d-189cdea16a39\" (UID: \"b72a9a3f-1065-40e4-a82d-189cdea16a39\") "
Oct 30 01:34:35.347136 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.347094    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-openshift-ca" pod "b72a9a3f-1065-40e4-a82d-189cdea16a39" (UID: "b72a9a3f-1065-40e4-a82d-189cdea16a39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:35.349184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.349143    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72a9a3f-1065-40e4-a82d-189cdea16a39-kube-api-access-m74vp" pod "b72a9a3f-1065-40e4-a82d-189cdea16a39" (UID: "b72a9a3f-1065-40e4-a82d-189cdea16a39"). InnerVolumeSpecName "kube-api-access-m74vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:35.349314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.349223    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-xks-ca" pod "b72a9a3f-1065-40e4-a82d-189cdea16a39" (UID: "b72a9a3f-1065-40e4-a82d-189cdea16a39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:35.448387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.448334    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m74vp\" (UniqueName: \"kubernetes.io/projected/b72a9a3f-1065-40e4-a82d-189cdea16a39-kube-api-access-m74vp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:35.448387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.448378    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:35.448387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:35.448398    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72a9a3f-1065-40e4-a82d-189cdea16a39-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:34:36.024480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:36.024094    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb" event={"ID":"b72a9a3f-1065-40e4-a82d-189cdea16a39","Type":"ContainerDied","Data":"5c082cb11d0441d6d51e3d9a20d2d91abe0d3480cba34ec7a04d77079e70643c"}
Oct 30 01:34:36.024480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:36.024167    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c082cb11d0441d6d51e3d9a20d2d91abe0d3480cba34ec7a04d77079e70643c"
Oct 30 01:34:36.024480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:36.024239    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"
Oct 30 01:34:36.170296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:36.170248    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:36.170838 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:36.170316    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:37.169716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:37.169679    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body=
Oct 30 01:34:37.169904 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:37.169748    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused"
Oct 30 01:34:39.103260 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:34:39.103224    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:34:39.171427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:39.171389    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:34:39.171617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:39.171452    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:34:40.212675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:40.212625    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:34:40.213220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:40.212716    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:34:41.214024 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:41.213984    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:34:41.214559 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:41.214060    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:34:42.255679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:42.255624    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:34:42.256221 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:42.255725    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:34:42.647675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:42.647588    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:42.648048 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:42.648026    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:42.648954 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:42.648935    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr"
Oct 30 01:34:43.889122 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.889082    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"]
Oct 30 01:34:43.889556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.889546    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b72a9a3f-1065-40e4-a82d-189cdea16a39" containerName="revoke-keys"
Oct 30 01:34:43.889604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.889560    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72a9a3f-1065-40e4-a82d-189cdea16a39" containerName="revoke-keys"
Oct 30 01:34:43.889670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.889660    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72a9a3f-1065-40e4-a82d-189cdea16a39" containerName="revoke-keys"
Oct 30 01:34:43.893054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.893036    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.923125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.923086    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"]
Oct 30 01:34:43.944945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.944907    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945031    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945130    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945198    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945254    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945303    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945344 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945331    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945356    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:43.945662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:43.945515    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcq9j\" (UniqueName: \"kubernetes.io/projected/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-kube-api-access-dcq9j\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.046598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046562    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.046819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046604    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.046819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046651    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.046819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046677    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.046819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046723    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.046819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046749    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.046819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046773    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.047193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046840    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcq9j\" (UniqueName: \"kubernetes.io/projected/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-kube-api-access-dcq9j\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.047193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.046915    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.047193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.047151    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.047355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.047190    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.047355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.047256    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.047583 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.047557    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.047801 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.047777    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.049103 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.049078    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.049410 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.049395    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.072285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.072256    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.072563 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.072544    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcq9j\" (UniqueName: \"kubernetes.io/projected/9a10bb9b-ade0-4b26-9b51-8133ffad51ff-kube-api-access-dcq9j\") pod \"maas-default-gateway-openshift-default-67bf986f99-pc85x\" (UID: \"9a10bb9b-ade0-4b26-9b51-8133ffad51ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.207679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.207643    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:44.526153 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.526116    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"]
Oct 30 01:34:44.528215 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:34:44.528185    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a10bb9b_ade0_4b26_9b51_8133ffad51ff.slice/crio-e82fd5ced32015910b198aa6ffa7e8f16f9722fe0ec3ca06611a6174cce2b985 WatchSource:0}: Error finding container e82fd5ced32015910b198aa6ffa7e8f16f9722fe0ec3ca06611a6174cce2b985: Status 404 returned error can't find the container with id e82fd5ced32015910b198aa6ffa7e8f16f9722fe0ec3ca06611a6174cce2b985
Oct 30 01:34:44.530645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.530610    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:34:44.530761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.530682    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:34:44.530761 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:44.530712    5104 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 01:34:45.108144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:45.108099    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" event={"ID":"9a10bb9b-ade0-4b26-9b51-8133ffad51ff","Type":"ContainerStarted","Data":"f6205c6236bdd62640522adc59e3f5acc52f648f6064eace484e446be146b0c8"}
Oct 30 01:34:45.108687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:45.108206    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" event={"ID":"9a10bb9b-ade0-4b26-9b51-8133ffad51ff","Type":"ContainerStarted","Data":"e82fd5ced32015910b198aa6ffa7e8f16f9722fe0ec3ca06611a6174cce2b985"}
Oct 30 01:34:45.162764 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:45.162700    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podStartSLOduration=2.162681892 podStartE2EDuration="2.162681892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:45.160404348 +0000 UTC m=+3126.708094844" watchObservedRunningTime="2026-10-30 01:34:45.162681892 +0000 UTC m=+3126.710372389"
Oct 30 01:34:45.208090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:45.208050    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:45.209884 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:45.209858    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:45.209942 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:45.209914    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:46.208924 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:46.208885    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:46.209363 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:46.208941    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:47.021864 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:47.021825    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:34:47.208847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:47.208811    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:47.209015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:47.208874    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:48.211634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:48.211586    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:48.216775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:48.212186    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:49.212297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:49.212254    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:49.212973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:49.212924    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:50.213482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:50.210958    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:50.213482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:50.211076    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:51.208210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:51.208162    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:51.208426 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:51.208230    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:52.208885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:52.208844    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused" start-of-body=
Oct 30 01:34:52.209320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:52.208900    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": dial tcp 10.132.1.181:15021: connect: connection refused"
Oct 30 01:34:54.208444 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:54.208385    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:34:54.209020 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:54.208452    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:34:55.249633 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.249593    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:34:55.250068 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.249653    5104 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:34:55.428368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.428333    5104 scope.go:122] "RemoveContainer" containerID="4e73fd99447efaa5e200d1a14b47fa6fe6834cf1ea4af50687e6210e4bae02cb"
Oct 30 01:34:55.439100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.439060    5104 scope.go:122] "RemoveContainer" containerID="01d748627b54ec0665ab150e6b4abb80ced7990709469e91361f9fee43c27ff2"
Oct 30 01:34:55.449865 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.449838    5104 scope.go:122] "RemoveContainer" containerID="20948c049f7c19b4db8146b617a783ab428dbf0bec7aa3657edffc2742dcc4da"
Oct 30 01:34:55.451752 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.451727    5104 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:55.452848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.452830    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:55.453556 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.453542    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x"
Oct 30 01:34:55.459868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.459845    5104 scope.go:122] "RemoveContainer" containerID="a7775e58d15aee2d57c3363b92bb0077b7da51e5b5097198eeab05b382007522"
Oct 30 01:34:55.469007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.468985    5104 scope.go:122] "RemoveContainer" containerID="d1e2db4f7aa14f14a737031b2067258c17936d7bedbf91fc58e667ce720ab961"
Oct 30 01:34:55.477856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.477831    5104 scope.go:122] "RemoveContainer" containerID="717f29e467c198132137e05719b6c81fb126ee023726e81e3c7a3d4cedd3a830"
Oct 30 01:34:55.489543 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.489524    5104 scope.go:122] "RemoveContainer" containerID="db20aca251cd0c17d0c4a0beeda68ba418773023a39f5e7dcf51c9134eb989c0"
Oct 30 01:34:55.498232 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:55.498194    5104 scope.go:122] "RemoveContainer" containerID="be5d57cd7eb0be6a2797837630b677a442b9f52b29a428245035749670807319"
Oct 30 01:34:57.027394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:57.027357    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"]
Oct 30 01:34:57.039021 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:57.038989    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d046b7-q62q6"]
Oct 30 01:34:57.041867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:57.041836    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"]
Oct 30 01:34:57.044046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:57.044015    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5c72e7-xt5c7"]
Oct 30 01:34:58.575872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.575837    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a" path="/var/lib/kubelet/pods/3f32d0a2-eab4-40ac-9f2a-c27d4bc9414a/volumes"
Oct 30 01:34:58.576285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.576259    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858f65ae-f849-4d29-9d85-865aa30e4352" path="/var/lib/kubelet/pods/858f65ae-f849-4d29-9d85-865aa30e4352/volumes"
Oct 30 01:34:58.919480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.919425    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"]
Oct 30 01:34:58.925439 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.925418    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:58.940276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.940256    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:34:58.940433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.940417    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:34:58.940505 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.940437    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:34:58.951910 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.951877    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:34:58.954525 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:58.954503    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"]
Oct 30 01:34:59.116245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.116199    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.116245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.116254    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.116541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.116271    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bzv\" (UniqueName: \"kubernetes.io/projected/ccb3ee48-39c8-493b-9711-22136649bbc6-kube-api-access-p4bzv\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.217579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.217488    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.217579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.217534    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.217579 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.217551    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bzv\" (UniqueName: \"kubernetes.io/projected/ccb3ee48-39c8-493b-9711-22136649bbc6-kube-api-access-p4bzv\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.218271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.218243    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.219878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.219859    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.237132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.237109    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bzv\" (UniqueName: \"kubernetes.io/projected/ccb3ee48-39c8-493b-9711-22136649bbc6-kube-api-access-p4bzv\") pod \"maas-api-revoke-sub-046e1343-clvxp\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.239912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.239894    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:34:59.544610 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:34:59.544581    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"]
Oct 30 01:34:59.547689 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:34:59.547657    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb3ee48_39c8_493b_9711_22136649bbc6.slice/crio-f39492c2f7fbf19027cab531b2a337a8371081f1cd074998bb7665943525fcc3 WatchSource:0}: Error finding container f39492c2f7fbf19027cab531b2a337a8371081f1cd074998bb7665943525fcc3: Status 404 returned error can't find the container with id f39492c2f7fbf19027cab531b2a337a8371081f1cd074998bb7665943525fcc3
Oct 30 01:35:00.257681 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:00.257634    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp" event={"ID":"ccb3ee48-39c8-493b-9711-22136649bbc6","Type":"ContainerStarted","Data":"1732e6553bad0e799d7b0de602bf2807bf3fb741a707dd95fc1f360f41d9622d"}
Oct 30 01:35:00.258181 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:00.257704    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp" event={"ID":"ccb3ee48-39c8-493b-9711-22136649bbc6","Type":"ContainerStarted","Data":"f39492c2f7fbf19027cab531b2a337a8371081f1cd074998bb7665943525fcc3"}
Oct 30 01:35:00.295220 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:00.295170    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp" podStartSLOduration=2.295157262 podStartE2EDuration="2.295157262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:00.294390781 +0000 UTC m=+3141.842081276" watchObservedRunningTime="2026-10-30 01:35:00.295157262 +0000 UTC m=+3141.842847757"
Oct 30 01:35:01.269966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:01.269929    5104 generic.go:356] "Generic (PLEG): container finished" podID="ccb3ee48-39c8-493b-9711-22136649bbc6" containerID="1732e6553bad0e799d7b0de602bf2807bf3fb741a707dd95fc1f360f41d9622d" exitCode=0
Oct 30 01:35:01.270388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:01.269983    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp" event={"ID":"ccb3ee48-39c8-493b-9711-22136649bbc6","Type":"ContainerDied","Data":"1732e6553bad0e799d7b0de602bf2807bf3fb741a707dd95fc1f360f41d9622d"}
Oct 30 01:35:02.425623 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.425594    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:35:02.448028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.447993    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-openshift-ca\") pod \"ccb3ee48-39c8-493b-9711-22136649bbc6\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") "
Oct 30 01:35:02.448199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.448032    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb3ee48-39c8-493b-9711-22136649bbc6-kube-api-access-p4bzv\" (UniqueName: \"kubernetes.io/projected/ccb3ee48-39c8-493b-9711-22136649bbc6-kube-api-access-p4bzv\") pod \"ccb3ee48-39c8-493b-9711-22136649bbc6\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") "
Oct 30 01:35:02.448199 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.448077    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-xks-ca\") pod \"ccb3ee48-39c8-493b-9711-22136649bbc6\" (UID: \"ccb3ee48-39c8-493b-9711-22136649bbc6\") "
Oct 30 01:35:02.448353 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.448331    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-openshift-ca" pod "ccb3ee48-39c8-493b-9711-22136649bbc6" (UID: "ccb3ee48-39c8-493b-9711-22136649bbc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:02.448442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.448419    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:02.450264 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.450241    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-xks-ca" pod "ccb3ee48-39c8-493b-9711-22136649bbc6" (UID: "ccb3ee48-39c8-493b-9711-22136649bbc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:02.450403 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.450287    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb3ee48-39c8-493b-9711-22136649bbc6-kube-api-access-p4bzv" pod "ccb3ee48-39c8-493b-9711-22136649bbc6" (UID: "ccb3ee48-39c8-493b-9711-22136649bbc6"). InnerVolumeSpecName "kube-api-access-p4bzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:02.549044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.548953    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb3ee48-39c8-493b-9711-22136649bbc6-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:02.549044 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:02.548987    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bzv\" (UniqueName: \"kubernetes.io/projected/ccb3ee48-39c8-493b-9711-22136649bbc6-kube-api-access-p4bzv\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:03.283925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:03.283892    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"
Oct 30 01:35:03.284115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:03.283927    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp" event={"ID":"ccb3ee48-39c8-493b-9711-22136649bbc6","Type":"ContainerDied","Data":"f39492c2f7fbf19027cab531b2a337a8371081f1cd074998bb7665943525fcc3"}
Oct 30 01:35:03.284115 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:03.283978    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f39492c2f7fbf19027cab531b2a337a8371081f1cd074998bb7665943525fcc3"
Oct 30 01:35:09.104800 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:35:09.104762    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:35:09.239646 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.239605    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"]
Oct 30 01:35:09.240395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.240335    5104 cpu_manager.go:414] "containerMap: removing container" podUID="ccb3ee48-39c8-493b-9711-22136649bbc6" containerName="revoke-keys"
Oct 30 01:35:09.240395 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.240361    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb3ee48-39c8-493b-9711-22136649bbc6" containerName="revoke-keys"
Oct 30 01:35:09.240667 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.240525    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb3ee48-39c8-493b-9711-22136649bbc6" containerName="revoke-keys"
Oct 30 01:35:09.243993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.243969    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.260390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.260361    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:35:09.271230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.271193    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:35:09.271230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.271196    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:35:09.271518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.271305    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:35:09.275517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.275490    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"]
Oct 30 01:35:09.310074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.309982    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xf8c\" (UniqueName: \"kubernetes.io/projected/b869940a-1443-420e-ae8e-577c66d5e875-kube-api-access-4xf8c\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.310327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.310087    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.310327 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.310141    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.410803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.410715    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xf8c\" (UniqueName: \"kubernetes.io/projected/b869940a-1443-420e-ae8e-577c66d5e875-kube-api-access-4xf8c\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.410803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.410770    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.411052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.410836    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.411473 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.411426    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.413354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.413331    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.435519 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.435484    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xf8c\" (UniqueName: \"kubernetes.io/projected/b869940a-1443-420e-ae8e-577c66d5e875-kube-api-access-4xf8c\") pod \"maas-api-revoke-sub-d3fba056-tmfln\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.557560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.557521    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:09.858404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:09.858376    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"]
Oct 30 01:35:09.861169 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:35:09.861126    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb869940a_1443_420e_ae8e_577c66d5e875.slice/crio-e3adfa11016c6764a1ee3e73959b67fb8f8c2155e6967ebffa068162c86d2f08 WatchSource:0}: Error finding container e3adfa11016c6764a1ee3e73959b67fb8f8c2155e6967ebffa068162c86d2f08: Status 404 returned error can't find the container with id e3adfa11016c6764a1ee3e73959b67fb8f8c2155e6967ebffa068162c86d2f08
Oct 30 01:35:10.149710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:10.149607    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:35:10.333259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:10.333219    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln" event={"ID":"b869940a-1443-420e-ae8e-577c66d5e875","Type":"ContainerStarted","Data":"0a3995cb6b4e6b641d32020b001f9855e29d889ed29984eaaf3d55c298a3f02e"}
Oct 30 01:35:10.333469 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:10.333288    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln" event={"ID":"b869940a-1443-420e-ae8e-577c66d5e875","Type":"ContainerStarted","Data":"e3adfa11016c6764a1ee3e73959b67fb8f8c2155e6967ebffa068162c86d2f08"}
Oct 30 01:35:10.376452 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:10.376381    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln" podStartSLOduration=1.37636151 podStartE2EDuration="1.37636151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:10.372658904 +0000 UTC m=+3151.920349401" watchObservedRunningTime="2026-10-30 01:35:10.37636151 +0000 UTC m=+3151.924052007"
Oct 30 01:35:11.345423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:11.345372    5104 generic.go:356] "Generic (PLEG): container finished" podID="b869940a-1443-420e-ae8e-577c66d5e875" containerID="0a3995cb6b4e6b641d32020b001f9855e29d889ed29984eaaf3d55c298a3f02e" exitCode=0
Oct 30 01:35:11.346673 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:11.345602    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln" event={"ID":"b869940a-1443-420e-ae8e-577c66d5e875","Type":"ContainerDied","Data":"0a3995cb6b4e6b641d32020b001f9855e29d889ed29984eaaf3d55c298a3f02e"}
Oct 30 01:35:13.132051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.118664    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"]
Oct 30 01:35:13.134912 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.134875    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3422aa3e-pjq9w"]
Oct 30 01:35:13.289692 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.289655    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:13.303108 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.303060    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b869940a-1443-420e-ae8e-577c66d5e875-kube-api-access-4xf8c\" (UniqueName: \"kubernetes.io/projected/b869940a-1443-420e-ae8e-577c66d5e875-kube-api-access-4xf8c\") pod \"b869940a-1443-420e-ae8e-577c66d5e875\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") "
Oct 30 01:35:13.303446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.303418    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-xks-ca\") pod \"b869940a-1443-420e-ae8e-577c66d5e875\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") "
Oct 30 01:35:13.303690 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.303670    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-openshift-ca\") pod \"b869940a-1443-420e-ae8e-577c66d5e875\" (UID: \"b869940a-1443-420e-ae8e-577c66d5e875\") "
Oct 30 01:35:13.308721 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.308688    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-openshift-ca" pod "b869940a-1443-420e-ae8e-577c66d5e875" (UID: "b869940a-1443-420e-ae8e-577c66d5e875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:13.325484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.323947    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-xks-ca" pod "b869940a-1443-420e-ae8e-577c66d5e875" (UID: "b869940a-1443-420e-ae8e-577c66d5e875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:13.331648 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.331609    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b869940a-1443-420e-ae8e-577c66d5e875-kube-api-access-4xf8c" pod "b869940a-1443-420e-ae8e-577c66d5e875" (UID: "b869940a-1443-420e-ae8e-577c66d5e875"). InnerVolumeSpecName "kube-api-access-4xf8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:13.461550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.416350    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xf8c\" (UniqueName: \"kubernetes.io/projected/b869940a-1443-420e-ae8e-577c66d5e875-kube-api-access-4xf8c\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:13.462699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.462664    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:13.462938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.462916    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b869940a-1443-420e-ae8e-577c66d5e875-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:13.466799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.463790    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln" event={"ID":"b869940a-1443-420e-ae8e-577c66d5e875","Type":"ContainerDied","Data":"e3adfa11016c6764a1ee3e73959b67fb8f8c2155e6967ebffa068162c86d2f08"}
Oct 30 01:35:13.466799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.463850    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3adfa11016c6764a1ee3e73959b67fb8f8c2155e6967ebffa068162c86d2f08"
Oct 30 01:35:13.466799 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:13.463919    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"
Oct 30 01:35:14.576595 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:14.576554    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121c1ac7-2e84-485a-883f-43db564126c2" path="/var/lib/kubelet/pods/121c1ac7-2e84-485a-883f-43db564126c2/volumes"
Oct 30 01:35:20.458656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:20.458619    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:35:25.004850 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:25.004812    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:35:26.051567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:26.035242    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"]
Oct 30 01:35:26.058851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:26.058810    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386e3c17-tdzgq"]
Oct 30 01:35:26.596098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:26.596049    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430994cb-e380-4b7b-9f56-4bdec2053601" path="/var/lib/kubelet/pods/430994cb-e380-4b7b-9f56-4bdec2053601/volumes"
Oct 30 01:35:27.034664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:27.034618    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"]
Oct 30 01:35:27.056492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:27.045774    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139ef9af-mdm9l"]
Oct 30 01:35:28.576385 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:28.576342    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a06c24-ebaf-43d1-bc51-3046fb9d6c04" path="/var/lib/kubelet/pods/36a06c24-ebaf-43d1-bc51-3046fb9d6c04/volumes"
Oct 30 01:35:30.702879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.702844    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"]
Oct 30 01:35:30.704091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.704052    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b869940a-1443-420e-ae8e-577c66d5e875" containerName="revoke-keys"
Oct 30 01:35:30.704091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.704080    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b869940a-1443-420e-ae8e-577c66d5e875" containerName="revoke-keys"
Oct 30 01:35:30.704315 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.704199    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b869940a-1443-420e-ae8e-577c66d5e875" containerName="revoke-keys"
Oct 30 01:35:30.708965 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.708944    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:30.737656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.737619    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:35:30.737854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.737656    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:35:30.738600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.738552    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"]
Oct 30 01:35:30.816889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.816859    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:35:30.817110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.816956    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:35:30.899875 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.899833    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:30.900111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.899905    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8mvl\" (UniqueName: \"kubernetes.io/projected/4172dbec-b8da-4384-9546-90e199daf07f-kube-api-access-w8mvl\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:30.900111 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:30.899979    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.000775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.000694    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8mvl\" (UniqueName: \"kubernetes.io/projected/4172dbec-b8da-4384-9546-90e199daf07f-kube-api-access-w8mvl\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.000775 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.000744    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.001110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.000805    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.001406 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.001386    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.003259 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.003231    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.019388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.019359    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8mvl\" (UniqueName: \"kubernetes.io/projected/4172dbec-b8da-4384-9546-90e199daf07f-kube-api-access-w8mvl\") pod \"maas-api-revoke-sub-ae594979-jxj6j\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.022160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.022140    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:31.319146 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.319115    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"]
Oct 30 01:35:31.321388 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:35:31.321356    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4172dbec_b8da_4384_9546_90e199daf07f.slice/crio-01c12336378b5f1a0102fa0e00b0ffee2df7d814e5d9a1cbbf7cc7da0ff2a2d6 WatchSource:0}: Error finding container 01c12336378b5f1a0102fa0e00b0ffee2df7d814e5d9a1cbbf7cc7da0ff2a2d6: Status 404 returned error can't find the container with id 01c12336378b5f1a0102fa0e00b0ffee2df7d814e5d9a1cbbf7cc7da0ff2a2d6
Oct 30 01:35:31.612450 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.612413    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j" event={"ID":"4172dbec-b8da-4384-9546-90e199daf07f","Type":"ContainerStarted","Data":"e135ebbaaae1c6e137f64143ece70dff2c4a3148fac7ee27b609ff895fd09255"}
Oct 30 01:35:31.612664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.612500    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j" event={"ID":"4172dbec-b8da-4384-9546-90e199daf07f","Type":"ContainerStarted","Data":"01c12336378b5f1a0102fa0e00b0ffee2df7d814e5d9a1cbbf7cc7da0ff2a2d6"}
Oct 30 01:35:31.649361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:31.649305    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j" podStartSLOduration=1.6492913599999999 podStartE2EDuration="1.64929136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:31.648239948 +0000 UTC m=+3173.195930445" watchObservedRunningTime="2026-10-30 01:35:31.64929136 +0000 UTC m=+3173.196981856"
Oct 30 01:35:32.620769 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:32.620731    5104 generic.go:356] "Generic (PLEG): container finished" podID="4172dbec-b8da-4384-9546-90e199daf07f" containerID="e135ebbaaae1c6e137f64143ece70dff2c4a3148fac7ee27b609ff895fd09255" exitCode=0
Oct 30 01:35:32.621195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:32.620812    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j" event={"ID":"4172dbec-b8da-4384-9546-90e199daf07f","Type":"ContainerDied","Data":"e135ebbaaae1c6e137f64143ece70dff2c4a3148fac7ee27b609ff895fd09255"}
Oct 30 01:35:33.774657 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.774631    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:33.825872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.825836    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-xks-ca\") pod \"4172dbec-b8da-4384-9546-90e199daf07f\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") "
Oct 30 01:35:33.826127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.825895    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-openshift-ca\") pod \"4172dbec-b8da-4384-9546-90e199daf07f\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") "
Oct 30 01:35:33.826127 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.825936    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4172dbec-b8da-4384-9546-90e199daf07f-kube-api-access-w8mvl\" (UniqueName: \"kubernetes.io/projected/4172dbec-b8da-4384-9546-90e199daf07f-kube-api-access-w8mvl\") pod \"4172dbec-b8da-4384-9546-90e199daf07f\" (UID: \"4172dbec-b8da-4384-9546-90e199daf07f\") "
Oct 30 01:35:33.826391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.826359    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-openshift-ca" pod "4172dbec-b8da-4384-9546-90e199daf07f" (UID: "4172dbec-b8da-4384-9546-90e199daf07f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:33.828225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.828192    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-xks-ca" pod "4172dbec-b8da-4384-9546-90e199daf07f" (UID: "4172dbec-b8da-4384-9546-90e199daf07f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:33.828225 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.828213    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4172dbec-b8da-4384-9546-90e199daf07f-kube-api-access-w8mvl" pod "4172dbec-b8da-4384-9546-90e199daf07f" (UID: "4172dbec-b8da-4384-9546-90e199daf07f"). InnerVolumeSpecName "kube-api-access-w8mvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:33.926880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.926847    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:33.926880 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.926883    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172dbec-b8da-4384-9546-90e199daf07f-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:33.927117 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:33.926893    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8mvl\" (UniqueName: \"kubernetes.io/projected/4172dbec-b8da-4384-9546-90e199daf07f-kube-api-access-w8mvl\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:34.635436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:34.635405    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"
Oct 30 01:35:34.635643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:34.635441    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j" event={"ID":"4172dbec-b8da-4384-9546-90e199daf07f","Type":"ContainerDied","Data":"01c12336378b5f1a0102fa0e00b0ffee2df7d814e5d9a1cbbf7cc7da0ff2a2d6"}
Oct 30 01:35:34.635643 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:34.635512    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c12336378b5f1a0102fa0e00b0ffee2df7d814e5d9a1cbbf7cc7da0ff2a2d6"
Oct 30 01:35:39.104602 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:35:39.104558    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:35:42.112121 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:42.112083    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:35:47.176514 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:47.176471    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:35:47.355056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:47.354572    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-j726m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:35:47.355056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:47.354640    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-j726m" podUID="cee3fb42-b55e-4887-b109-efbe113cef93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:35:52.065508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.065393    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"]
Oct 30 01:35:52.066030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.066009    5104 cpu_manager.go:414] "containerMap: removing container" podUID="4172dbec-b8da-4384-9546-90e199daf07f" containerName="revoke-keys"
Oct 30 01:35:52.066086 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.066032    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4172dbec-b8da-4384-9546-90e199daf07f" containerName="revoke-keys"
Oct 30 01:35:52.066145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.066136    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="4172dbec-b8da-4384-9546-90e199daf07f" containerName="revoke-keys"
Oct 30 01:35:52.069677 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.069656    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.088837 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.088798    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"]
Oct 30 01:35:52.099496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.097661    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:35:52.099496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.097678    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:35:52.099496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.097713    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:35:52.099496 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.097786    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:35:52.184295 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.184255    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.184524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.184429    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.184524 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.184484    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpsh4\" (UniqueName: \"kubernetes.io/projected/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-kube-api-access-vpsh4\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.285918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.285875    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.286107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.285994    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.286107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.286026    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpsh4\" (UniqueName: \"kubernetes.io/projected/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-kube-api-access-vpsh4\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.286678 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.286653    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.288501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.288443    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.304629 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.304593    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpsh4\" (UniqueName: \"kubernetes.io/projected/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-kube-api-access-vpsh4\") pod \"maas-api-revoke-sub-a4aef7d7-vf4dv\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.384158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.384073    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:52.676818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.676793    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"]
Oct 30 01:35:52.679407 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:35:52.679377    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27b7c46_e419_4dfa_bdf4_e17e3f1cb552.slice/crio-ff1a7fde06893acaec6b1c5f166ca1e5dacc9808ece601dc8949ccc91dbfe993 WatchSource:0}: Error finding container ff1a7fde06893acaec6b1c5f166ca1e5dacc9808ece601dc8949ccc91dbfe993: Status 404 returned error can't find the container with id ff1a7fde06893acaec6b1c5f166ca1e5dacc9808ece601dc8949ccc91dbfe993
Oct 30 01:35:52.799096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:52.799058    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv" event={"ID":"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552","Type":"ContainerStarted","Data":"ff1a7fde06893acaec6b1c5f166ca1e5dacc9808ece601dc8949ccc91dbfe993"}
Oct 30 01:35:53.806819 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:53.806783    5104 generic.go:356] "Generic (PLEG): container finished" podID="c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" containerID="3d03e28ba104f86c0c4db3ed6666191a13818e0614ad30f8e51beb84bfe9774d" exitCode=0
Oct 30 01:35:53.807197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:53.806870    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv" event={"ID":"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552","Type":"ContainerDied","Data":"3d03e28ba104f86c0c4db3ed6666191a13818e0614ad30f8e51beb84bfe9774d"}
Oct 30 01:35:54.962170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:54.962139    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:55.011751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.011710    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-openshift-ca\") pod \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") "
Oct 30 01:35:55.011950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.011764    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-xks-ca\") pod \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") "
Oct 30 01:35:55.011950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.011934    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-kube-api-access-vpsh4\" (UniqueName: \"kubernetes.io/projected/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-kube-api-access-vpsh4\") pod \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\" (UID: \"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552\") "
Oct 30 01:35:55.012150 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.012123    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-openshift-ca" pod "c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" (UID: "c27b7c46-e419-4dfa-bdf4-e17e3f1cb552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:55.012338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.012306    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:55.014106 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.014058    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-xks-ca" pod "c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" (UID: "c27b7c46-e419-4dfa-bdf4-e17e3f1cb552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:55.014640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.014612    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-kube-api-access-vpsh4" pod "c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" (UID: "c27b7c46-e419-4dfa-bdf4-e17e3f1cb552"). InnerVolumeSpecName "kube-api-access-vpsh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:55.113691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.113603    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpsh4\" (UniqueName: \"kubernetes.io/projected/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-kube-api-access-vpsh4\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:55.113691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.113637    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:35:55.551165 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.551129    5104 scope.go:122] "RemoveContainer" containerID="9b74ad9989b7351a1a76c7124c105704657d7f2c3c355c3ccbb3ab583f348f68"
Oct 30 01:35:55.563489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.563442    5104 scope.go:122] "RemoveContainer" containerID="582ef7131a32bc43c957f702b43424d760bf158a4b418f6d63a262185ac1936d"
Oct 30 01:35:55.578175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.578144    5104 scope.go:122] "RemoveContainer" containerID="11f4f92ef6d9ef98b67827d967b6fdc9d7a80f411732480043cdb793b2e7e52f"
Oct 30 01:35:55.592252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.592222    5104 scope.go:122] "RemoveContainer" containerID="257619c1e40e0bcb9a6d3c1c273bd328f3c8f22ebeb29feff8a89b326d379a71"
Oct 30 01:35:55.602212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.602185    5104 scope.go:122] "RemoveContainer" containerID="b70b2c2a327ea7b9c2f80d4bc22da9b74b33e13060513d6549adaec5dbf25c84"
Oct 30 01:35:55.611251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.611231    5104 scope.go:122] "RemoveContainer" containerID="06fc7cb739ef626c6afadfa5493a0afad35bb28fa4c27504776bd4369fb26d77"
Oct 30 01:35:55.620822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.620800    5104 scope.go:122] "RemoveContainer" containerID="3cdcf68463b329b239df88847702a702539a0af9e947183cf94f680d60e7da34"
Oct 30 01:35:55.633030 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.633005    5104 scope.go:122] "RemoveContainer" containerID="04826a8dbda1c905c25ce8cb01dbbfac69d018f651b1d88213315544535aa509"
Oct 30 01:35:55.641817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.641793    5104 scope.go:122] "RemoveContainer" containerID="a7d059ef4ba13cd176684c2a881ccbce9e7edfd143102dd9334cff6f993ab367"
Oct 30 01:35:55.650576 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.650555    5104 scope.go:122] "RemoveContainer" containerID="f951d69962db72409b7f33dd06a6befa47bd13f3da72a4cfad3132b32e8e4450"
Oct 30 01:35:55.820733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.820643    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"
Oct 30 01:35:55.820733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.820676    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv" event={"ID":"c27b7c46-e419-4dfa-bdf4-e17e3f1cb552","Type":"ContainerDied","Data":"ff1a7fde06893acaec6b1c5f166ca1e5dacc9808ece601dc8949ccc91dbfe993"}
Oct 30 01:35:55.820733 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:35:55.820723    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1a7fde06893acaec6b1c5f166ca1e5dacc9808ece601dc8949ccc91dbfe993"
Oct 30 01:36:03.232194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:03.232154    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:36:07.603614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:07.603559    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:36:09.262481 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:36:09.260233    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:36:12.072112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.072078    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"]
Oct 30 01:36:12.072585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.072569    5104 cpu_manager.go:414] "containerMap: removing container" podUID="c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" containerName="revoke-keys"
Oct 30 01:36:12.072585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.072586    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" containerName="revoke-keys"
Oct 30 01:36:12.072719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.072700    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" containerName="revoke-keys"
Oct 30 01:36:12.076027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.076006    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.095032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.094994    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"]
Oct 30 01:36:12.103499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.103468    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:36:12.103499 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.103487    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:36:12.103735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.103515    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:36:12.103735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.103647    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:36:12.151863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.151826    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.151863 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.151868    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgdk\" (UniqueName: \"kubernetes.io/projected/f415eead-8f39-4961-8d9d-a2ab819662d5-kube-api-access-xbgdk\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.152089 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.151982    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.253479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.253409    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.253679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.253489    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgdk\" (UniqueName: \"kubernetes.io/projected/f415eead-8f39-4961-8d9d-a2ab819662d5-kube-api-access-xbgdk\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.253679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.253531    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.254235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.254215    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.256011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.255990    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.274515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.274488    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgdk\" (UniqueName: \"kubernetes.io/projected/f415eead-8f39-4961-8d9d-a2ab819662d5-kube-api-access-xbgdk\") pod \"maas-api-revoke-sub-79805ca9-c9r97\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.390607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.390524    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:12.685243 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:12.684930    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"]
Oct 30 01:36:12.687192 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:36:12.687157    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf415eead_8f39_4961_8d9d_a2ab819662d5.slice/crio-685e0e6a9fbd3115a645165cb786309c4353883380b833035c859e60684d0084 WatchSource:0}: Error finding container 685e0e6a9fbd3115a645165cb786309c4353883380b833035c859e60684d0084: Status 404 returned error can't find the container with id 685e0e6a9fbd3115a645165cb786309c4353883380b833035c859e60684d0084
Oct 30 01:36:13.026482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:13.026419    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97" event={"ID":"f415eead-8f39-4961-8d9d-a2ab819662d5","Type":"ContainerStarted","Data":"778bc6d0ad43c43bdb31287d037074736d3ec66dad1a137b6fc363c1ede7ba28"}
Oct 30 01:36:13.026665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:13.026509    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97" event={"ID":"f415eead-8f39-4961-8d9d-a2ab819662d5","Type":"ContainerStarted","Data":"685e0e6a9fbd3115a645165cb786309c4353883380b833035c859e60684d0084"}
Oct 30 01:36:13.064918 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:13.064865    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97" podStartSLOduration=1.064850261 podStartE2EDuration="1.064850261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:13.063630259 +0000 UTC m=+3214.611320755" watchObservedRunningTime="2026-10-30 01:36:13.064850261 +0000 UTC m=+3214.612540757"
Oct 30 01:36:14.034854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:14.034816    5104 generic.go:356] "Generic (PLEG): container finished" podID="f415eead-8f39-4961-8d9d-a2ab819662d5" containerID="778bc6d0ad43c43bdb31287d037074736d3ec66dad1a137b6fc363c1ede7ba28" exitCode=0
Oct 30 01:36:14.035273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:14.034903    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97" event={"ID":"f415eead-8f39-4961-8d9d-a2ab819662d5","Type":"ContainerDied","Data":"778bc6d0ad43c43bdb31287d037074736d3ec66dad1a137b6fc363c1ede7ba28"}
Oct 30 01:36:15.190853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.190826    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:15.286170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.286127    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-xks-ca\") pod \"f415eead-8f39-4961-8d9d-a2ab819662d5\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") "
Oct 30 01:36:15.286392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.286189    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f415eead-8f39-4961-8d9d-a2ab819662d5-kube-api-access-xbgdk\" (UniqueName: \"kubernetes.io/projected/f415eead-8f39-4961-8d9d-a2ab819662d5-kube-api-access-xbgdk\") pod \"f415eead-8f39-4961-8d9d-a2ab819662d5\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") "
Oct 30 01:36:15.286392 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.286291    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-openshift-ca\") pod \"f415eead-8f39-4961-8d9d-a2ab819662d5\" (UID: \"f415eead-8f39-4961-8d9d-a2ab819662d5\") "
Oct 30 01:36:15.286684 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.286656    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-openshift-ca" pod "f415eead-8f39-4961-8d9d-a2ab819662d5" (UID: "f415eead-8f39-4961-8d9d-a2ab819662d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:15.286946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.286921    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:15.288585 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.288552    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f415eead-8f39-4961-8d9d-a2ab819662d5-kube-api-access-xbgdk" pod "f415eead-8f39-4961-8d9d-a2ab819662d5" (UID: "f415eead-8f39-4961-8d9d-a2ab819662d5"). InnerVolumeSpecName "kube-api-access-xbgdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:15.288746 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.288673    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-xks-ca" pod "f415eead-8f39-4961-8d9d-a2ab819662d5" (UID: "f415eead-8f39-4961-8d9d-a2ab819662d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:15.388194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.388157    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f415eead-8f39-4961-8d9d-a2ab819662d5-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:15.388194 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:15.388190    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbgdk\" (UniqueName: \"kubernetes.io/projected/f415eead-8f39-4961-8d9d-a2ab819662d5-kube-api-access-xbgdk\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:16.048811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:16.048775    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"
Oct 30 01:36:16.048811 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:16.048806    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97" event={"ID":"f415eead-8f39-4961-8d9d-a2ab819662d5","Type":"ContainerDied","Data":"685e0e6a9fbd3115a645165cb786309c4353883380b833035c859e60684d0084"}
Oct 30 01:36:16.049046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:16.048893    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="685e0e6a9fbd3115a645165cb786309c4353883380b833035c859e60684d0084"
Oct 30 01:36:23.237709 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:23.237668    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:36:28.261951 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:28.261903    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:36:28.693645 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:28.693602    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:36:28.694007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:28.693969    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:36:33.106512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.106447    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"]
Oct 30 01:36:33.106955 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.106940    5104 cpu_manager.go:414] "containerMap: removing container" podUID="f415eead-8f39-4961-8d9d-a2ab819662d5" containerName="revoke-keys"
Oct 30 01:36:33.107007 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.106956    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f415eead-8f39-4961-8d9d-a2ab819662d5" containerName="revoke-keys"
Oct 30 01:36:33.107056 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.107026    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="f415eead-8f39-4961-8d9d-a2ab819662d5" containerName="revoke-keys"
Oct 30 01:36:33.110381 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.110361    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.138306 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.138269    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:36:33.138577 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.138294    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:36:33.138688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.138293    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:36:33.138688 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.138339    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:36:33.142180 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.142138    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"]
Oct 30 01:36:33.212144 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.212109    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.212326 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.212240    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krf5w\" (UniqueName: \"kubernetes.io/projected/59c8455e-dd34-44d5-bab0-04ed95ab12bf-kube-api-access-krf5w\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.212384 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.212364    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.313868 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.313827    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.314100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.313999    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krf5w\" (UniqueName: \"kubernetes.io/projected/59c8455e-dd34-44d5-bab0-04ed95ab12bf-kube-api-access-krf5w\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.314182 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.314100    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.314632 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.314608    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.316497 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.316471    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.333323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.333285    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krf5w\" (UniqueName: \"kubernetes.io/projected/59c8455e-dd34-44d5-bab0-04ed95ab12bf-kube-api-access-krf5w\") pod \"maas-api-revoke-sub-0b7e8edb-pfjnr\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.429129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.429098    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:33.726749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:33.726720    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"]
Oct 30 01:36:33.728989 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:36:33.728962    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c8455e_dd34_44d5_bab0_04ed95ab12bf.slice/crio-667d634b288fb8d4a5332ce0b36c42b532a376aae891c4c0a45bdd549564683a WatchSource:0}: Error finding container 667d634b288fb8d4a5332ce0b36c42b532a376aae891c4c0a45bdd549564683a: Status 404 returned error can't find the container with id 667d634b288fb8d4a5332ce0b36c42b532a376aae891c4c0a45bdd549564683a
Oct 30 01:36:34.278240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:34.278155    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr" event={"ID":"59c8455e-dd34-44d5-bab0-04ed95ab12bf","Type":"ContainerStarted","Data":"64c11260bc70873618c5e9830066485f3db4318c2b17622cde781ef7e2c542d3"}
Oct 30 01:36:34.278240 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:34.278208    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr" event={"ID":"59c8455e-dd34-44d5-bab0-04ed95ab12bf","Type":"ContainerStarted","Data":"667d634b288fb8d4a5332ce0b36c42b532a376aae891c4c0a45bdd549564683a"}
Oct 30 01:36:34.318027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:34.317971    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr" podStartSLOduration=1.317955908 podStartE2EDuration="1.317955908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:34.316633148 +0000 UTC m=+3235.864323657" watchObservedRunningTime="2026-10-30 01:36:34.317955908 +0000 UTC m=+3235.865646440"
Oct 30 01:36:35.285784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:35.285745    5104 generic.go:356] "Generic (PLEG): container finished" podID="59c8455e-dd34-44d5-bab0-04ed95ab12bf" containerID="64c11260bc70873618c5e9830066485f3db4318c2b17622cde781ef7e2c542d3" exitCode=0
Oct 30 01:36:35.286223 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:35.285827    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr" event={"ID":"59c8455e-dd34-44d5-bab0-04ed95ab12bf","Type":"ContainerDied","Data":"64c11260bc70873618c5e9830066485f3db4318c2b17622cde781ef7e2c542d3"}
Oct 30 01:36:36.434651 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.434620    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:36.548098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.548011    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-openshift-ca\") pod \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") "
Oct 30 01:36:36.548261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.548098    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-xks-ca\") pod \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") "
Oct 30 01:36:36.548261 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.548182    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c8455e-dd34-44d5-bab0-04ed95ab12bf-kube-api-access-krf5w\" (UniqueName: \"kubernetes.io/projected/59c8455e-dd34-44d5-bab0-04ed95ab12bf-kube-api-access-krf5w\") pod \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\" (UID: \"59c8455e-dd34-44d5-bab0-04ed95ab12bf\") "
Oct 30 01:36:36.548391 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.548368    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-openshift-ca" pod "59c8455e-dd34-44d5-bab0-04ed95ab12bf" (UID: "59c8455e-dd34-44d5-bab0-04ed95ab12bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:36.548552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.548532    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:36.550273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.550243    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-xks-ca" pod "59c8455e-dd34-44d5-bab0-04ed95ab12bf" (UID: "59c8455e-dd34-44d5-bab0-04ed95ab12bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:36.550394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.550351    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c8455e-dd34-44d5-bab0-04ed95ab12bf-kube-api-access-krf5w" pod "59c8455e-dd34-44d5-bab0-04ed95ab12bf" (UID: "59c8455e-dd34-44d5-bab0-04ed95ab12bf"). InnerVolumeSpecName "kube-api-access-krf5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:36.649691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.649659    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8455e-dd34-44d5-bab0-04ed95ab12bf-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:36.649691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:36.649688    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krf5w\" (UniqueName: \"kubernetes.io/projected/59c8455e-dd34-44d5-bab0-04ed95ab12bf-kube-api-access-krf5w\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:37.299379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:37.299341    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"
Oct 30 01:36:37.299611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:37.299328    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr" event={"ID":"59c8455e-dd34-44d5-bab0-04ed95ab12bf","Type":"ContainerDied","Data":"667d634b288fb8d4a5332ce0b36c42b532a376aae891c4c0a45bdd549564683a"}
Oct 30 01:36:37.299611 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:37.299514    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667d634b288fb8d4a5332ce0b36c42b532a376aae891c4c0a45bdd549564683a"
Oct 30 01:36:39.103081 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:36:39.103044    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:36:43.800140 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.800094    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"]
Oct 30 01:36:43.800879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.800853    5104 cpu_manager.go:414] "containerMap: removing container" podUID="59c8455e-dd34-44d5-bab0-04ed95ab12bf" containerName="revoke-keys"
Oct 30 01:36:43.800983 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.800881    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c8455e-dd34-44d5-bab0-04ed95ab12bf" containerName="revoke-keys"
Oct 30 01:36:43.801113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.801096    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c8455e-dd34-44d5-bab0-04ed95ab12bf" containerName="revoke-keys"
Oct 30 01:36:43.805052 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.805025    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:43.822003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.821970    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"]
Oct 30 01:36:43.831066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.830165    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:36:43.831066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.830173    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:36:43.831066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.830234    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:36:43.831066 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.830259    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:36:43.919807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.919771    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhz79\" (UniqueName: \"kubernetes.io/projected/bb104712-57ab-426d-b1a3-2a125da5bb99-kube-api-access-xhz79\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:43.920027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.919859    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:43.920027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:43.919882    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.020818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.020782    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.020818 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.020820    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.021090 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.020934    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhz79\" (UniqueName: \"kubernetes.io/projected/bb104712-57ab-426d-b1a3-2a125da5bb99-kube-api-access-xhz79\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.021582 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.021556    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.023416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.023393    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.039985 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.039948    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhz79\" (UniqueName: \"kubernetes.io/projected/bb104712-57ab-426d-b1a3-2a125da5bb99-kube-api-access-xhz79\") pod \"maas-api-revoke-sub-3d4156f1-m742l\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.119479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.119376    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:44.400208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.400116    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:36:44.420113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:44.420076    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"]
Oct 30 01:36:44.422025 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:36:44.421996    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb104712_57ab_426d_b1a3_2a125da5bb99.slice/crio-a34de543a79f4a89fcceb5c175dd4cfaaf31a35f44d6d8c033b0616d06171f2d WatchSource:0}: Error finding container a34de543a79f4a89fcceb5c175dd4cfaaf31a35f44d6d8c033b0616d06171f2d: Status 404 returned error can't find the container with id a34de543a79f4a89fcceb5c175dd4cfaaf31a35f44d6d8c033b0616d06171f2d
Oct 30 01:36:45.351175 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:45.351134    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l" event={"ID":"bb104712-57ab-426d-b1a3-2a125da5bb99","Type":"ContainerStarted","Data":"44ffa4d71b7634413d092291ff083f6ef5212e77d2e8b0011153bde58f599e64"}
Oct 30 01:36:45.351618 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:45.351217    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l" event={"ID":"bb104712-57ab-426d-b1a3-2a125da5bb99","Type":"ContainerStarted","Data":"a34de543a79f4a89fcceb5c175dd4cfaaf31a35f44d6d8c033b0616d06171f2d"}
Oct 30 01:36:45.389682 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:45.389626    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l" podStartSLOduration=2.389612518 podStartE2EDuration="2.389612518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:45.388891723 +0000 UTC m=+3246.936582219" watchObservedRunningTime="2026-10-30 01:36:45.389612518 +0000 UTC m=+3246.937303013"
Oct 30 01:36:46.359218 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:46.359182    5104 generic.go:356] "Generic (PLEG): container finished" podID="bb104712-57ab-426d-b1a3-2a125da5bb99" containerID="44ffa4d71b7634413d092291ff083f6ef5212e77d2e8b0011153bde58f599e64" exitCode=0
Oct 30 01:36:46.359687 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:46.359272    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l" event={"ID":"bb104712-57ab-426d-b1a3-2a125da5bb99","Type":"ContainerDied","Data":"44ffa4d71b7634413d092291ff083f6ef5212e77d2e8b0011153bde58f599e64"}
Oct 30 01:36:47.515547 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.515509    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:47.547143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.547107    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-openshift-ca\") pod \"bb104712-57ab-426d-b1a3-2a125da5bb99\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") "
Oct 30 01:36:47.547338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.547179    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-xks-ca\") pod \"bb104712-57ab-426d-b1a3-2a125da5bb99\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") "
Oct 30 01:36:47.547338 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.547217    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb104712-57ab-426d-b1a3-2a125da5bb99-kube-api-access-xhz79\" (UniqueName: \"kubernetes.io/projected/bb104712-57ab-426d-b1a3-2a125da5bb99-kube-api-access-xhz79\") pod \"bb104712-57ab-426d-b1a3-2a125da5bb99\" (UID: \"bb104712-57ab-426d-b1a3-2a125da5bb99\") "
Oct 30 01:36:47.547602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.547573    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-openshift-ca" pod "bb104712-57ab-426d-b1a3-2a125da5bb99" (UID: "bb104712-57ab-426d-b1a3-2a125da5bb99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:47.549397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.549373    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-xks-ca" pod "bb104712-57ab-426d-b1a3-2a125da5bb99" (UID: "bb104712-57ab-426d-b1a3-2a125da5bb99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:47.549397 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.549397    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb104712-57ab-426d-b1a3-2a125da5bb99-kube-api-access-xhz79" pod "bb104712-57ab-426d-b1a3-2a125da5bb99" (UID: "bb104712-57ab-426d-b1a3-2a125da5bb99"). InnerVolumeSpecName "kube-api-access-xhz79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:47.648751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.648656    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:47.649058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.649038    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb104712-57ab-426d-b1a3-2a125da5bb99-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:47.649168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:47.649155    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhz79\" (UniqueName: \"kubernetes.io/projected/bb104712-57ab-426d-b1a3-2a125da5bb99-kube-api-access-xhz79\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:36:48.374517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:48.374483    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"
Oct 30 01:36:48.374517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:48.374489    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l" event={"ID":"bb104712-57ab-426d-b1a3-2a125da5bb99","Type":"ContainerDied","Data":"a34de543a79f4a89fcceb5c175dd4cfaaf31a35f44d6d8c033b0616d06171f2d"}
Oct 30 01:36:48.374749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:48.374570    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34de543a79f4a89fcceb5c175dd4cfaaf31a35f44d6d8c033b0616d06171f2d"
Oct 30 01:36:55.017650 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.017606    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:36:55.716325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.716274    5104 scope.go:122] "RemoveContainer" containerID="3ba5200f9e3380de1280268f8050a04648224dc4cf8268df2d479bfcf408ca13"
Oct 30 01:36:55.733017 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.732990    5104 scope.go:122] "RemoveContainer" containerID="7f5c22bbe90de3de376773bd3c76886770b2b1651b4dc4426cd68c28ef5b9817"
Oct 30 01:36:55.746963 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.746931    5104 scope.go:122] "RemoveContainer" containerID="c70d6b0cb52ee477d2c3cbefbc1681c54ae409b57dca0e9777d271260a171d09"
Oct 30 01:36:55.760747 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.760718    5104 scope.go:122] "RemoveContainer" containerID="101636d0832e3c540306037b924daadb43d39532b29b9d3d12e368dbfc8546bf"
Oct 30 01:36:55.774354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.774326    5104 scope.go:122] "RemoveContainer" containerID="0e3dba7217e5141ad91725f08e919d0dd83dd140fa8a7232f6a82958abc17c1d"
Oct 30 01:36:55.787988 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.787956    5104 scope.go:122] "RemoveContainer" containerID="8e6ad3933e6055f0d925f5b038fe9c858607a978d62abab0175d15c37a18d2ed"
Oct 30 01:36:55.801803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:36:55.801685    5104 scope.go:122] "RemoveContainer" containerID="198c8597a5499cc245c595dec6dcdf732fec7532209435164b7c1ab7127b6118"
Oct 30 01:37:00.527134 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:00.527100    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:37:03.028074 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:03.028030    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"]
Oct 30 01:37:03.039028 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:03.038993    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"]
Oct 30 01:37:03.041975 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:03.041947    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17374661-wctvp"]
Oct 30 01:37:03.044188 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:03.044162    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df878348-j9fjh"]
Oct 30 01:37:04.575930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:04.575894    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db7542b-893d-4d17-85e2-ccadaf949201" path="/var/lib/kubelet/pods/6db7542b-893d-4d17-85e2-ccadaf949201/volumes"
Oct 30 01:37:04.576350 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:04.576336    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abdc9c46-8122-45a4-9080-cba7ab8bd44e" path="/var/lib/kubelet/pods/abdc9c46-8122-45a4-9080-cba7ab8bd44e/volumes"
Oct 30 01:37:04.961360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:04.961315    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"]
Oct 30 01:37:04.972340 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:04.972303    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m742l"]
Oct 30 01:37:05.017325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.017290    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"]
Oct 30 01:37:05.017814 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.017800    5104 cpu_manager.go:414] "containerMap: removing container" podUID="bb104712-57ab-426d-b1a3-2a125da5bb99" containerName="revoke-keys"
Oct 30 01:37:05.017862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.017815    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb104712-57ab-426d-b1a3-2a125da5bb99" containerName="revoke-keys"
Oct 30 01:37:05.017909 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.017881    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb104712-57ab-426d-b1a3-2a125da5bb99" containerName="revoke-keys"
Oct 30 01:37:05.022948 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.022929    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.037308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.037284    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:05.037308 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.037292    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:05.039139 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.039113    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"]
Oct 30 01:37:05.075737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.075697    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4fg\" (UniqueName: \"kubernetes.io/projected/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-kube-api-access-nw4fg\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.075969 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.075873    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.076042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.075973    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.095245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.095214    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:37:05.095400 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.095384    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:37:05.177277 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.177243    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4fg\" (UniqueName: \"kubernetes.io/projected/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-kube-api-access-nw4fg\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.177572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.177405    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.177572 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.177526    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.178093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.178062    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.179877 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.179851    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.197045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.197018    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4fg\" (UniqueName: \"kubernetes.io/projected/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-kube-api-access-nw4fg\") pod \"maas-api-revoke-sub-3d4156f1-5lm52\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.336336 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.336248    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:05.630094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:05.630066    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"]
Oct 30 01:37:05.631814 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:37:05.631783    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7858ab4_4ad3_4f70_9252_4cfdde809cfb.slice/crio-5e9c4d0fa03f790293479121a49d96721fd19226d29f88fb2910033d1297636c WatchSource:0}: Error finding container 5e9c4d0fa03f790293479121a49d96721fd19226d29f88fb2910033d1297636c: Status 404 returned error can't find the container with id 5e9c4d0fa03f790293479121a49d96721fd19226d29f88fb2910033d1297636c
Oct 30 01:37:06.554429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:06.554398    5104 generic.go:356] "Generic (PLEG): container finished" podID="a7858ab4-4ad3-4f70-9252-4cfdde809cfb" containerID="8747e9b844aeb434604d6f7315befee23222656536f95baf590e6dadc41f0db4" exitCode=0
Oct 30 01:37:06.554608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:06.554500    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52" event={"ID":"a7858ab4-4ad3-4f70-9252-4cfdde809cfb","Type":"ContainerDied","Data":"8747e9b844aeb434604d6f7315befee23222656536f95baf590e6dadc41f0db4"}
Oct 30 01:37:06.554608 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:06.554564    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52" event={"ID":"a7858ab4-4ad3-4f70-9252-4cfdde809cfb","Type":"ContainerStarted","Data":"5e9c4d0fa03f790293479121a49d96721fd19226d29f88fb2910033d1297636c"}
Oct 30 01:37:06.575244 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:06.575211    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb104712-57ab-426d-b1a3-2a125da5bb99" path="/var/lib/kubelet/pods/bb104712-57ab-426d-b1a3-2a125da5bb99/volumes"
Oct 30 01:37:07.708712 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.703377    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:07.806037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.805997    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-openshift-ca\") pod \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") "
Oct 30 01:37:07.806226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.806146    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-kube-api-access-nw4fg\" (UniqueName: \"kubernetes.io/projected/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-kube-api-access-nw4fg\") pod \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") "
Oct 30 01:37:07.806226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.806183    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-xks-ca\") pod \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\" (UID: \"a7858ab4-4ad3-4f70-9252-4cfdde809cfb\") "
Oct 30 01:37:07.806446 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.806420    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-openshift-ca" pod "a7858ab4-4ad3-4f70-9252-4cfdde809cfb" (UID: "a7858ab4-4ad3-4f70-9252-4cfdde809cfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:37:07.806599 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.806582    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:37:07.808328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.808302    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-xks-ca" pod "a7858ab4-4ad3-4f70-9252-4cfdde809cfb" (UID: "a7858ab4-4ad3-4f70-9252-4cfdde809cfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:37:07.808414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.808382    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-kube-api-access-nw4fg" pod "a7858ab4-4ad3-4f70-9252-4cfdde809cfb" (UID: "a7858ab4-4ad3-4f70-9252-4cfdde809cfb"). InnerVolumeSpecName "kube-api-access-nw4fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:37:07.907270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.907237    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw4fg\" (UniqueName: \"kubernetes.io/projected/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-kube-api-access-nw4fg\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:37:07.907270 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:07.907269    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7858ab4-4ad3-4f70-9252-4cfdde809cfb-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:37:08.568511 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:08.568472    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52" event={"ID":"a7858ab4-4ad3-4f70-9252-4cfdde809cfb","Type":"ContainerDied","Data":"5e9c4d0fa03f790293479121a49d96721fd19226d29f88fb2910033d1297636c"}
Oct 30 01:37:08.568716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:08.568557    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9c4d0fa03f790293479121a49d96721fd19226d29f88fb2910033d1297636c"
Oct 30 01:37:08.568716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:08.568494    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"
Oct 30 01:37:09.105313 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:37:09.105263    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:37:22.094031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.093978    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"]
Oct 30 01:37:22.103215 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.103180    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5lm52"]
Oct 30 01:37:22.138492 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.138438    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"]
Oct 30 01:37:22.139235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.139206    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a7858ab4-4ad3-4f70-9252-4cfdde809cfb" containerName="revoke-keys"
Oct 30 01:37:22.139378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.139239    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7858ab4-4ad3-4f70-9252-4cfdde809cfb" containerName="revoke-keys"
Oct 30 01:37:22.139378 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.139364    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7858ab4-4ad3-4f70-9252-4cfdde809cfb" containerName="revoke-keys"
Oct 30 01:37:22.142932 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.142909    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.158676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.158645    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:37:22.158676 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.158660    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:22.158870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.158663    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:37:22.158870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.158663    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:22.161917 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.161890    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"]
Oct 30 01:37:22.251411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.251371    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcxp\" (UniqueName: \"kubernetes.io/projected/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-kube-api-access-hfcxp\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.251411 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.251416    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.251664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.251441    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.352658 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.352557    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.352852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.352720    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcxp\" (UniqueName: \"kubernetes.io/projected/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-kube-api-access-hfcxp\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.352852 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.352769    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.353414 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.353387    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.354922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.354903    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.370986 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.370950    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcxp\" (UniqueName: \"kubernetes.io/projected/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-kube-api-access-hfcxp\") pod \"maas-api-revoke-sub-3d4156f1-mr6cv\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.458698 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.458660    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:22.579756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.579710    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7858ab4-4ad3-4f70-9252-4cfdde809cfb" path="/var/lib/kubelet/pods/a7858ab4-4ad3-4f70-9252-4cfdde809cfb/volumes"
Oct 30 01:37:22.755655 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:22.755625    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"]
Oct 30 01:37:22.757752 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:37:22.757720    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f5e68d_2f1e_4610_ab24_851ac7c8e82d.slice/crio-07a1cbf9537d5406ab19788cbf6e2a6c2529b310cc368d7596798317c76a1549 WatchSource:0}: Error finding container 07a1cbf9537d5406ab19788cbf6e2a6c2529b310cc368d7596798317c76a1549: Status 404 returned error can't find the container with id 07a1cbf9537d5406ab19788cbf6e2a6c2529b310cc368d7596798317c76a1549
Oct 30 01:37:23.658095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:23.658059    5104 generic.go:356] "Generic (PLEG): container finished" podID="d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" containerID="96a0acc2ae4b1209b95b64e132ac01ca18dc824382f1ac9a487995dc284bdfea" exitCode=0
Oct 30 01:37:23.658591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:23.658152    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv" event={"ID":"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d","Type":"ContainerDied","Data":"96a0acc2ae4b1209b95b64e132ac01ca18dc824382f1ac9a487995dc284bdfea"}
Oct 30 01:37:23.658591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:23.658226    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv" event={"ID":"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d","Type":"ContainerStarted","Data":"07a1cbf9537d5406ab19788cbf6e2a6c2529b310cc368d7596798317c76a1549"}
Oct 30 01:37:24.027834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.027792    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"]
Oct 30 01:37:24.031540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.031510    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"]
Oct 30 01:37:24.039037 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.039004    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be992ab-gb6lb"]
Oct 30 01:37:24.041586 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.041559    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09c2efb-8p7tq"]
Oct 30 01:37:24.577012 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.576978    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a3b34a-572b-4c88-b459-14d860ed649b" path="/var/lib/kubelet/pods/54a3b34a-572b-4c88-b459-14d860ed649b/volumes"
Oct 30 01:37:24.577404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.577391    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b50207f-830b-4526-93b0-001008700eda" path="/var/lib/kubelet/pods/7b50207f-830b-4526-93b0-001008700eda/volumes"
Oct 30 01:37:24.822882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.822854    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:24.978977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.978930    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-xks-ca\") pod \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") "
Oct 30 01:37:24.979184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.979006    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-kube-api-access-hfcxp\" (UniqueName: \"kubernetes.io/projected/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-kube-api-access-hfcxp\") pod \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") "
Oct 30 01:37:24.979184 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.979082    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-openshift-ca\") pod \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\" (UID: \"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d\") "
Oct 30 01:37:24.979501 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.979427    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-openshift-ca" pod "d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" (UID: "d4f5e68d-2f1e-4610-ab24-851ac7c8e82d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:37:24.981209 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.981180    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-xks-ca" pod "d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" (UID: "d4f5e68d-2f1e-4610-ab24-851ac7c8e82d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:37:24.981318 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:24.981251    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-kube-api-access-hfcxp" pod "d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" (UID: "d4f5e68d-2f1e-4610-ab24-851ac7c8e82d"). InnerVolumeSpecName "kube-api-access-hfcxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:37:25.080404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:25.080363    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfcxp\" (UniqueName: \"kubernetes.io/projected/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-kube-api-access-hfcxp\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:37:25.080404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:25.080395    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:37:25.080404 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:25.080410    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:37:25.671339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:25.671300    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"
Oct 30 01:37:25.671339 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:25.671325    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv" event={"ID":"d4f5e68d-2f1e-4610-ab24-851ac7c8e82d","Type":"ContainerDied","Data":"07a1cbf9537d5406ab19788cbf6e2a6c2529b310cc368d7596798317c76a1549"}
Oct 30 01:37:25.671593 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:25.671369    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a1cbf9537d5406ab19788cbf6e2a6c2529b310cc368d7596798317c76a1549"
Oct 30 01:37:39.104437 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:37:39.104392    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:37:39.743438 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:39.743402    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:37:39.745129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:39.745104    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:37:39.751946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:39.751923    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:37:39.753417 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:39.753395    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:37:42.694596 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.694558    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"]
Oct 30 01:37:42.695174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.695154    5104 cpu_manager.go:414] "containerMap: removing container" podUID="d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" containerName="revoke-keys"
Oct 30 01:37:42.695239 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.695175    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" containerName="revoke-keys"
Oct 30 01:37:42.695292 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.695268    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" containerName="revoke-keys"
Oct 30 01:37:42.698757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.698738    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.715696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.715642    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-37ba51-serving-cert\""
Oct 30 01:37:42.715696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.715647    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:42.715696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.715663    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:42.727348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.727303    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:37:42.727671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.727642    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nss4l\""
Oct 30 01:37:42.730987 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.730955    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"]
Oct 30 01:37:42.752622 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.752589    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.753011 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.752976    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-metrics-certs\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.753141 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.753025    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779135a0-b70b-4ab3-b254-1a913191b805-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.753207 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.753176    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncrt\" (UniqueName: \"kubernetes.io/projected/779135a0-b70b-4ab3-b254-1a913191b805-kube-api-access-vncrt\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.854756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.854696    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-metrics-certs\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.854756 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.854762    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779135a0-b70b-4ab3-b254-1a913191b805-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.855046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.854834    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncrt\" (UniqueName: \"kubernetes.io/projected/779135a0-b70b-4ab3-b254-1a913191b805-kube-api-access-vncrt\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.855046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.854870    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.855046 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:37:42.855041    5104 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-serving-cert: secret "maas-api-e2e-worker-single-main-37ba51-serving-cert" not found
Oct 30 01:37:42.855229 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:37:42.855109    5104 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls podName:779135a0-b70b-4ab3-b254-1a913191b805 nodeName:}" failed. No retries permitted until 2026-10-30 01:37:43.355088167 +0000 UTC m=+3304.902778658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls") pod "maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z" (UID: "779135a0-b70b-4ab3-b254-1a913191b805") : secret "maas-api-e2e-worker-single-main-37ba51-serving-cert" not found
Oct 30 01:37:42.855844 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.855805    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779135a0-b70b-4ab3-b254-1a913191b805-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.858205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.858172    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-metrics-certs\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:42.874132 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:42.874102    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncrt\" (UniqueName: \"kubernetes.io/projected/779135a0-b70b-4ab3-b254-1a913191b805-kube-api-access-vncrt\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:43.084626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.084541    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"]
Oct 30 01:37:43.088639 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.088621    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.105708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.105679    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-37ba51\""
Oct 30 01:37:43.108945 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.108913    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"]
Oct 30 01:37:43.174479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.164249    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6f20578c-dockercfg-xfk8b\""
Oct 30 01:37:43.271101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.271062    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr5gd\" (UniqueName: \"kubernetes.io/projected/2be63367-6433-4fa1-94f7-781bf8330366-kube-api-access-qr5gd\") pod \"payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.271297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.271187    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be63367-6433-4fa1-94f7-781bf8330366-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.372251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.372152    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:43.372251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.372201    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr5gd\" (UniqueName: \"kubernetes.io/projected/2be63367-6433-4fa1-94f7-781bf8330366-kube-api-access-qr5gd\") pod \"payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.372531 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.372297    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be63367-6433-4fa1-94f7-781bf8330366-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.373101 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.373071    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be63367-6433-4fa1-94f7-781bf8330366-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.374828 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.374802    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:43.391187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.391161    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr5gd\" (UniqueName: \"kubernetes.io/projected/2be63367-6433-4fa1-94f7-781bf8330366-kube-api-access-qr5gd\") pod \"payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.403156 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.403131    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:43.618311 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.618277    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:43.713046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.712976    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"]
Oct 30 01:37:43.717427 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:37:43.717392    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be63367_6433_4fa1_94f7_781bf8330366.slice/crio-d5d438641985c06743ed2eb41e07fc16b1088e00f452ecbd941cd1b782e24c21 WatchSource:0}: Error finding container d5d438641985c06743ed2eb41e07fc16b1088e00f452ecbd941cd1b782e24c21: Status 404 returned error can't find the container with id d5d438641985c06743ed2eb41e07fc16b1088e00f452ecbd941cd1b782e24c21
Oct 30 01:37:43.796157 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.796124    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" event={"ID":"2be63367-6433-4fa1-94f7-781bf8330366","Type":"ContainerStarted","Data":"d5d438641985c06743ed2eb41e07fc16b1088e00f452ecbd941cd1b782e24c21"}
Oct 30 01:37:43.937062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:43.936950    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"]
Oct 30 01:37:43.939773 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:37:43.939739    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779135a0_b70b_4ab3_b254_1a913191b805.slice/crio-a4c47b7e1ef04478c5d6bc5622c09888fd33fe02728064d9e4ae8dd5425a109a WatchSource:0}: Error finding container a4c47b7e1ef04478c5d6bc5622c09888fd33fe02728064d9e4ae8dd5425a109a: Status 404 returned error can't find the container with id a4c47b7e1ef04478c5d6bc5622c09888fd33fe02728064d9e4ae8dd5425a109a
Oct 30 01:37:44.805428 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:44.805384    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z" event={"ID":"779135a0-b70b-4ab3-b254-1a913191b805","Type":"ContainerStarted","Data":"5a02ce3894d91d4b324968f9280033fa330007c6c514604ce996d5d8ac5e95ae"}
Oct 30 01:37:44.806027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:44.805524    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z" event={"ID":"779135a0-b70b-4ab3-b254-1a913191b805","Type":"ContainerStarted","Data":"a4c47b7e1ef04478c5d6bc5622c09888fd33fe02728064d9e4ae8dd5425a109a"}
Oct 30 01:37:44.806027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:44.805556    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:44.807394 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:44.807365    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" event={"ID":"2be63367-6433-4fa1-94f7-781bf8330366","Type":"ContainerStarted","Data":"6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d"}
Oct 30 01:37:44.807567 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:44.807518    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:44.847489 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:44.847404    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z" podStartSLOduration=2.426489279 podStartE2EDuration="2.8473848s" totalImagesPullingTime="420.895521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:44.843712421 +0000 UTC m=+3306.391402915" watchObservedRunningTime="2026-10-30 01:37:44.8473848 +0000 UTC m=+3306.395075299"
Oct 30 01:37:44.880866 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:44.880801    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" podStartSLOduration=1.880782417 podStartE2EDuration="1.880782417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:44.8803984 +0000 UTC m=+3306.428088895" watchObservedRunningTime="2026-10-30 01:37:44.880782417 +0000 UTC m=+3306.428472912"
Oct 30 01:37:46.532203 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:46.532149    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:37:50.821401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:50.821369    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:37:55.818600 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:55.818569    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:37:55.888617 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:55.888577    5104 scope.go:122] "RemoveContainer" containerID="5082cfc098d752e2062f9a1b87d218d2345038f0c559732c9303ffcaa05c2a78"
Oct 30 01:37:55.899936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:55.899908    5104 scope.go:122] "RemoveContainer" containerID="0a4926bad7aa7bff861203637d0fdec7077fab4f4aedd90fce138808f2269c26"
Oct 30 01:37:55.910640 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:55.910613    5104 scope.go:122] "RemoveContainer" containerID="16ca2e165bda7e9ec6349ba1d9bcb977f9d7c5d41a15096bb048791351e1de78"
Oct 30 01:37:55.921210 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:37:55.921177    5104 scope.go:122] "RemoveContainer" containerID="91e10102550c9972d83552d9ce2bd2c4bbcb02a6b82a983231671c4b22bdc129"
Oct 30 01:38:00.234190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:00.234151    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:38:09.104472 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:38:09.104415    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:38:14.033580 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:14.033536    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"]
Oct 30 01:38:14.044711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:14.044677    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2nq9t"]
Oct 30 01:38:14.576555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:14.576519    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d" path="/var/lib/kubelet/pods/60261e55-00cb-4c5e-b85c-bb0a6fdfdd0d/volumes"
Oct 30 01:38:15.789026 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:15.788987    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:38:33.036679 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:33.036641    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:38:39.114403 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:38:39.114357    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:38:43.268606 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:43.268564    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"]
Oct 30 01:38:43.769211 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:43.769173    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"]
Oct 30 01:38:43.769443 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:43.769408    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z" podUID="779135a0-b70b-4ab3-b254-1a913191b805" containerName="maas-api" containerID="cri-o://5a02ce3894d91d4b324968f9280033fa330007c6c514604ce996d5d8ac5e95ae" gracePeriod=30
Oct 30 01:38:44.221197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.221050    5104 generic.go:356] "Generic (PLEG): container finished" podID="779135a0-b70b-4ab3-b254-1a913191b805" containerID="5a02ce3894d91d4b324968f9280033fa330007c6c514604ce996d5d8ac5e95ae" exitCode=0
Oct 30 01:38:44.221197 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.221141    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z" event={"ID":"779135a0-b70b-4ab3-b254-1a913191b805","Type":"ContainerDied","Data":"5a02ce3894d91d4b324968f9280033fa330007c6c514604ce996d5d8ac5e95ae"}
Oct 30 01:38:44.288124 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.288092    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:38:44.443964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.443928    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls\") pod \"779135a0-b70b-4ab3-b254-1a913191b805\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") "
Oct 30 01:38:44.444191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.444020    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779135a0-b70b-4ab3-b254-1a913191b805-kube-api-access-vncrt\" (UniqueName: \"kubernetes.io/projected/779135a0-b70b-4ab3-b254-1a913191b805-kube-api-access-vncrt\") pod \"779135a0-b70b-4ab3-b254-1a913191b805\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") "
Oct 30 01:38:44.444191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.444079    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-metrics-certs\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-metrics-certs\") pod \"779135a0-b70b-4ab3-b254-1a913191b805\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") "
Oct 30 01:38:44.444191 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.444120    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779135a0-b70b-4ab3-b254-1a913191b805-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779135a0-b70b-4ab3-b254-1a913191b805-openshift-service-ca\") pod \"779135a0-b70b-4ab3-b254-1a913191b805\" (UID: \"779135a0-b70b-4ab3-b254-1a913191b805\") "
Oct 30 01:38:44.444612 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.444578    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779135a0-b70b-4ab3-b254-1a913191b805-openshift-service-ca" pod "779135a0-b70b-4ab3-b254-1a913191b805" (UID: "779135a0-b70b-4ab3-b254-1a913191b805"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:38:44.446370 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.446338    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779135a0-b70b-4ab3-b254-1a913191b805-kube-api-access-vncrt" pod "779135a0-b70b-4ab3-b254-1a913191b805" (UID: "779135a0-b70b-4ab3-b254-1a913191b805"). InnerVolumeSpecName "kube-api-access-vncrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:38:44.446807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.446781    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls" pod "779135a0-b70b-4ab3-b254-1a913191b805" (UID: "779135a0-b70b-4ab3-b254-1a913191b805"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:38:44.446807 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.446795    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-metrics-certs" pod "779135a0-b70b-4ab3-b254-1a913191b805" (UID: "779135a0-b70b-4ab3-b254-1a913191b805"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:38:44.545109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.545067    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-maas-api-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:38:44.545109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.545100    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vncrt\" (UniqueName: \"kubernetes.io/projected/779135a0-b70b-4ab3-b254-1a913191b805-kube-api-access-vncrt\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:38:44.545109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.545114    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/779135a0-b70b-4ab3-b254-1a913191b805-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:38:44.545427 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:44.545127    5104 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779135a0-b70b-4ab3-b254-1a913191b805-openshift-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:38:45.227773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:45.227733    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z" event={"ID":"779135a0-b70b-4ab3-b254-1a913191b805","Type":"ContainerDied","Data":"a4c47b7e1ef04478c5d6bc5622c09888fd33fe02728064d9e4ae8dd5425a109a"}
Oct 30 01:38:45.228096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:45.227779    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"
Oct 30 01:38:45.228096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:45.227825    5104 scope.go:122] "RemoveContainer" containerID="5a02ce3894d91d4b324968f9280033fa330007c6c514604ce996d5d8ac5e95ae"
Oct 30 01:38:45.278162 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:45.278121    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"]
Oct 30 01:38:45.289331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:45.289291    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-37ba51-597c46864d-62l9z"]
Oct 30 01:38:46.258348 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:46.258310    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:38:46.577251 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:46.577170    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779135a0-b70b-4ab3-b254-1a913191b805" path="/var/lib/kubelet/pods/779135a0-b70b-4ab3-b254-1a913191b805/volumes"
Oct 30 01:38:48.270006 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:48.269961    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" podUID="2be63367-6433-4fa1-94f7-781bf8330366" containerName="payload-pre-processing" containerID="cri-o://6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d" gracePeriod=25
Oct 30 01:38:54.386614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.386527    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"]
Oct 30 01:38:54.398193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.398158    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mr6cv"]
Oct 30 01:38:54.452782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.452739    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"]
Oct 30 01:38:54.453236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.453222    5104 cpu_manager.go:414] "containerMap: removing container" podUID="779135a0-b70b-4ab3-b254-1a913191b805" containerName="maas-api"
Oct 30 01:38:54.453288 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.453238    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779135a0-b70b-4ab3-b254-1a913191b805" containerName="maas-api"
Oct 30 01:38:54.453351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.453339    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="779135a0-b70b-4ab3-b254-1a913191b805" containerName="maas-api"
Oct 30 01:38:54.458076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.458055    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.473742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.473718    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:38:54.473742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.473714    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:38:54.473928 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.473740    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:38:54.484549 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.484520    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:38:54.488226 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.488193    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"]
Oct 30 01:38:54.560128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.560078    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7d6\" (UniqueName: \"kubernetes.io/projected/82870ec3-d036-4b87-94a8-12290a4ddc42-kube-api-access-qc7d6\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.560342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.560175    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.560342 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.560271    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.575956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.575925    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f5e68d-2f1e-4610-ab24-851ac7c8e82d" path="/var/lib/kubelet/pods/d4f5e68d-2f1e-4610-ab24-851ac7c8e82d/volumes"
Oct 30 01:38:54.661742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.661629    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7d6\" (UniqueName: \"kubernetes.io/projected/82870ec3-d036-4b87-94a8-12290a4ddc42-kube-api-access-qc7d6\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.661742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.661684    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.662076 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.661791    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.662520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.662494    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.664205 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.664179    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.682325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.682300    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7d6\" (UniqueName: \"kubernetes.io/projected/82870ec3-d036-4b87-94a8-12290a4ddc42-kube-api-access-qc7d6\") pod \"maas-api-revoke-sub-3d4156f1-zd2vb\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:54.770858 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:54.770822    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:55.077534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:55.077498    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"]
Oct 30 01:38:55.078189 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:38:55.078155    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82870ec3_d036_4b87_94a8_12290a4ddc42.slice/crio-91b4174847dce176eab40f7417beca1c902393aa114f506cc15caf1a3ef8eda3 WatchSource:0}: Error finding container 91b4174847dce176eab40f7417beca1c902393aa114f506cc15caf1a3ef8eda3: Status 404 returned error can't find the container with id 91b4174847dce176eab40f7417beca1c902393aa114f506cc15caf1a3ef8eda3
Oct 30 01:38:55.292964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:55.292926    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb" event={"ID":"82870ec3-d036-4b87-94a8-12290a4ddc42","Type":"ContainerStarted","Data":"91b4174847dce176eab40f7417beca1c902393aa114f506cc15caf1a3ef8eda3"}
Oct 30 01:38:55.817346 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:55.817302    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" podUID="2be63367-6433-4fa1-94f7-781bf8330366" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:38:55.968097 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:55.968066    5104 scope.go:122] "RemoveContainer" containerID="441d837c4f8cad6c2f8a8be17a7bcbefe95a88c5f250f38368048112bd768d80"
Oct 30 01:38:55.978128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:55.978101    5104 scope.go:122] "RemoveContainer" containerID="a7b772466eec5f44a7dbec3e68a303fda66b6422a556d632b8f20325d8727a8a"
Oct 30 01:38:56.027711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:56.027683    5104 scope.go:122] "RemoveContainer" containerID="1aa8df8b2972462589c6c254ecb48e0d3a382a16cf5443fca5dbe8ad45f40c16"
Oct 30 01:38:56.036665 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:56.036645    5104 scope.go:122] "RemoveContainer" containerID="a884d62b66d27200fe9e234a21804a78bf58c5dcf07ddc0ec15194fdb95ac04d"
Oct 30 01:38:56.044719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:56.044701    5104 scope.go:122] "RemoveContainer" containerID="772ec177ee16993e9e114c9e1f986cc278d228bc90f6594762e4fd93a52c7d19"
Oct 30 01:38:56.299636 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:56.299602    5104 generic.go:356] "Generic (PLEG): container finished" podID="82870ec3-d036-4b87-94a8-12290a4ddc42" containerID="6ed38de059a2235b78fe187b59cfb45f03cf64b3533a268e0ecb875a2053d06a" exitCode=0
Oct 30 01:38:56.299834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:56.299690    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb" event={"ID":"82870ec3-d036-4b87-94a8-12290a4ddc42","Type":"ContainerDied","Data":"6ed38de059a2235b78fe187b59cfb45f03cf64b3533a268e0ecb875a2053d06a"}
Oct 30 01:38:57.450316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.450293    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:57.488887 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.488853    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-xks-ca\") pod \"82870ec3-d036-4b87-94a8-12290a4ddc42\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") "
Oct 30 01:38:57.489083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.488908    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82870ec3-d036-4b87-94a8-12290a4ddc42-kube-api-access-qc7d6\" (UniqueName: \"kubernetes.io/projected/82870ec3-d036-4b87-94a8-12290a4ddc42-kube-api-access-qc7d6\") pod \"82870ec3-d036-4b87-94a8-12290a4ddc42\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") "
Oct 30 01:38:57.489083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.488928    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-openshift-ca\") pod \"82870ec3-d036-4b87-94a8-12290a4ddc42\" (UID: \"82870ec3-d036-4b87-94a8-12290a4ddc42\") "
Oct 30 01:38:57.489313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.489290    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-openshift-ca" pod "82870ec3-d036-4b87-94a8-12290a4ddc42" (UID: "82870ec3-d036-4b87-94a8-12290a4ddc42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:38:57.491010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.490984    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-xks-ca" pod "82870ec3-d036-4b87-94a8-12290a4ddc42" (UID: "82870ec3-d036-4b87-94a8-12290a4ddc42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:38:57.491159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.491137    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82870ec3-d036-4b87-94a8-12290a4ddc42-kube-api-access-qc7d6" pod "82870ec3-d036-4b87-94a8-12290a4ddc42" (UID: "82870ec3-d036-4b87-94a8-12290a4ddc42"). InnerVolumeSpecName "kube-api-access-qc7d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:38:57.590297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.590204    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:38:57.590297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.590236    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7d6\" (UniqueName: \"kubernetes.io/projected/82870ec3-d036-4b87-94a8-12290a4ddc42-kube-api-access-qc7d6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:38:57.590297 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:57.590246    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82870ec3-d036-4b87-94a8-12290a4ddc42-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:38:58.311886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:58.311842    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb" event={"ID":"82870ec3-d036-4b87-94a8-12290a4ddc42","Type":"ContainerDied","Data":"91b4174847dce176eab40f7417beca1c902393aa114f506cc15caf1a3ef8eda3"}
Oct 30 01:38:58.311886 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:58.311873    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"
Oct 30 01:38:58.312129 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:38:58.311934    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91b4174847dce176eab40f7417beca1c902393aa114f506cc15caf1a3ef8eda3"
Oct 30 01:39:04.038123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:04.038083    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"]
Oct 30 01:39:04.047770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:04.047742    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kpvhf"]
Oct 30 01:39:04.575529 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:04.575499    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b128c1ba-58de-401e-a359-8099d2de926b" path="/var/lib/kubelet/pods/b128c1ba-58de-401e-a359-8099d2de926b/volumes"
Oct 30 01:39:05.816937 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:05.816900    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" podUID="2be63367-6433-4fa1-94f7-781bf8330366" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:39:08.532751 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:08.532672    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:39:08.607283 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:08.607244    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be63367-6433-4fa1-94f7-781bf8330366-kube-api-access-qr5gd\" (UniqueName: \"kubernetes.io/projected/2be63367-6433-4fa1-94f7-781bf8330366-kube-api-access-qr5gd\") pod \"2be63367-6433-4fa1-94f7-781bf8330366\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") "
Oct 30 01:39:08.607526 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:08.607297    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be63367-6433-4fa1-94f7-781bf8330366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be63367-6433-4fa1-94f7-781bf8330366-plugins-config-volume\") pod \"2be63367-6433-4fa1-94f7-781bf8330366\" (UID: \"2be63367-6433-4fa1-94f7-781bf8330366\") "
Oct 30 01:39:08.607749 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:08.607723    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be63367-6433-4fa1-94f7-781bf8330366-plugins-config-volume" pod "2be63367-6433-4fa1-94f7-781bf8330366" (UID: "2be63367-6433-4fa1-94f7-781bf8330366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:39:08.609663 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:08.609635    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be63367-6433-4fa1-94f7-781bf8330366-kube-api-access-qr5gd" pod "2be63367-6433-4fa1-94f7-781bf8330366" (UID: "2be63367-6433-4fa1-94f7-781bf8330366"). InnerVolumeSpecName "kube-api-access-qr5gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:08.708821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:08.708787    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr5gd\" (UniqueName: \"kubernetes.io/projected/2be63367-6433-4fa1-94f7-781bf8330366-kube-api-access-qr5gd\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:39:08.708821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:08.708815    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be63367-6433-4fa1-94f7-781bf8330366-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:39:09.104196 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:39:09.104152    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:39:09.374885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.374789    5104 generic.go:356] "Generic (PLEG): container finished" podID="2be63367-6433-4fa1-94f7-781bf8330366" containerID="6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d" exitCode=0
Oct 30 01:39:09.374885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.374836    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"
Oct 30 01:39:09.374885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.374865    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" event={"ID":"2be63367-6433-4fa1-94f7-781bf8330366","Type":"ContainerDied","Data":"6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d"}
Oct 30 01:39:09.375228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.374921    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m" event={"ID":"2be63367-6433-4fa1-94f7-781bf8330366","Type":"ContainerDied","Data":"d5d438641985c06743ed2eb41e07fc16b1088e00f452ecbd941cd1b782e24c21"}
Oct 30 01:39:09.375228 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.374945    5104 scope.go:122] "RemoveContainer" containerID="6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d"
Oct 30 01:39:09.385597 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.385525    5104 scope.go:122] "RemoveContainer" containerID="6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d"
Oct 30 01:39:09.385857 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:39:09.385835    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d\": container with ID starting with 6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d not found: ID does not exist" containerID="6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d"
Oct 30 01:39:09.385922 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.385864    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d"} err="failed to get container status \"6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d\": rpc error: code = NotFound desc = could not find container \"6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d\": container with ID starting with 6ca0003facccaed1e68124aa043dab923373847193ad73c4c1e501d856abfd8d not found: ID does not exist"
Oct 30 01:39:09.426735 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.426702    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"]
Oct 30 01:39:09.437998 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:09.437968    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-37ba51-6fdd8zrk2m"]
Oct 30 01:39:10.575797 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:10.575765    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be63367-6433-4fa1-94f7-781bf8330366" path="/var/lib/kubelet/pods/2be63367-6433-4fa1-94f7-781bf8330366/volumes"
Oct 30 01:39:12.191355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.191314    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"]
Oct 30 01:39:12.191915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.191663    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj" podUID="b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" containerName="manager" containerID="cri-o://a089eaa8decf5de1c03ff0c970e8477d6bc60111104a95bd472d702f4c44af14" gracePeriod=10
Oct 30 01:39:12.396159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.396123    5104 generic.go:356] "Generic (PLEG): container finished" podID="b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" containerID="a089eaa8decf5de1c03ff0c970e8477d6bc60111104a95bd472d702f4c44af14" exitCode=0
Oct 30 01:39:12.396358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.396297    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj" event={"ID":"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4","Type":"ContainerDied","Data":"a089eaa8decf5de1c03ff0c970e8477d6bc60111104a95bd472d702f4c44af14"}
Oct 30 01:39:12.459540 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.459513    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 01:39:12.547399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.547361    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-extensions-socket-volume\") pod \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") "
Oct 30 01:39:12.547653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.547601    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-kube-api-access-dxdbz\" (UniqueName: \"kubernetes.io/projected/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-kube-api-access-dxdbz\") pod \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\" (UID: \"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4\") "
Oct 30 01:39:12.547803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.547779    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-extensions-socket-volume" pod "b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" (UID: "b1b8f43d-70bd-42d4-b1b0-283fa053f8f4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:12.547931 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.547913    5104 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-extensions-socket-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:39:12.549921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.549882    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-kube-api-access-dxdbz" pod "b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" (UID: "b1b8f43d-70bd-42d4-b1b0-283fa053f8f4"). InnerVolumeSpecName "kube-api-access-dxdbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:12.648925 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:12.648889    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdbz\" (UniqueName: \"kubernetes.io/projected/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4-kube-api-access-dxdbz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:39:13.403598 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:13.403563    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"
Oct 30 01:39:13.404054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:13.403594    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj" event={"ID":"b1b8f43d-70bd-42d4-b1b0-283fa053f8f4","Type":"ContainerDied","Data":"47ba89f388f7a291b739f86ffefac5f68fc93fad89887fc0cdb74db3fa200ee0"}
Oct 30 01:39:13.404054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:13.403654    5104 scope.go:122] "RemoveContainer" containerID="a089eaa8decf5de1c03ff0c970e8477d6bc60111104a95bd472d702f4c44af14"
Oct 30 01:39:13.469304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:13.469269    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"]
Oct 30 01:39:13.480695 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:13.480669    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-pc7pj"]
Oct 30 01:39:14.575258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:14.575222    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" path="/var/lib/kubelet/pods/b1b8f43d-70bd-42d4-b1b0-283fa053f8f4/volumes"
Oct 30 01:39:23.026878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:23.026838    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"]
Oct 30 01:39:23.038027 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:23.037998    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w456d"]
Oct 30 01:39:24.575399 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:24.575356    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e02772-efd4-494e-a0f6-3de43112e62b" path="/var/lib/kubelet/pods/c8e02772-efd4-494e-a0f6-3de43112e62b/volumes"
Oct 30 01:39:36.025802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:36.025756    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"]
Oct 30 01:39:36.036872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:36.036843    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r5qwb"]
Oct 30 01:39:36.575708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:36.575665    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72a9a3f-1065-40e4-a82d-189cdea16a39" path="/var/lib/kubelet/pods/b72a9a3f-1065-40e4-a82d-189cdea16a39/volumes"
Oct 30 01:39:39.103743 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:39:39.103707    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:39:56.087247 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:39:56.087209    5104 scope.go:122] "RemoveContainer" containerID="b0db68b56a05c4d507a9e809cc89643ea96c2563552f053b2bb6108f9bac9e9b"
Oct 30 01:40:03.030825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:03.030787    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"]
Oct 30 01:40:03.041628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:03.041591    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-clvxp"]
Oct 30 01:40:04.576091 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:04.576052    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb3ee48-39c8-493b-9711-22136649bbc6" path="/var/lib/kubelet/pods/ccb3ee48-39c8-493b-9711-22136649bbc6/volumes"
Oct 30 01:40:09.103548 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:40:09.103512    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:40:13.029561 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.029525    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"]
Oct 30 01:40:13.040862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.040799    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmfln"]
Oct 30 01:40:13.143848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.143813    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"]
Oct 30 01:40:13.144388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144357    5104 cpu_manager.go:414] "containerMap: removing container" podUID="82870ec3-d036-4b87-94a8-12290a4ddc42" containerName="revoke-keys"
Oct 30 01:40:13.144388 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144380    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82870ec3-d036-4b87-94a8-12290a4ddc42" containerName="revoke-keys"
Oct 30 01:40:13.144637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144404    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2be63367-6433-4fa1-94f7-781bf8330366" containerName="payload-pre-processing"
Oct 30 01:40:13.144637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144413    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be63367-6433-4fa1-94f7-781bf8330366" containerName="payload-pre-processing"
Oct 30 01:40:13.144637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144428    5104 cpu_manager.go:414] "containerMap: removing container" podUID="b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" containerName="manager"
Oct 30 01:40:13.144637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144434    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" containerName="manager"
Oct 30 01:40:13.144637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144548    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="82870ec3-d036-4b87-94a8-12290a4ddc42" containerName="revoke-keys"
Oct 30 01:40:13.144637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144562    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be63367-6433-4fa1-94f7-781bf8330366" containerName="payload-pre-processing"
Oct 30 01:40:13.144637 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.144569    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b8f43d-70bd-42d4-b1b0-283fa053f8f4" containerName="manager"
Oct 30 01:40:13.147812 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.147793    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.164908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.164888    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v6n6c\""
Oct 30 01:40:13.178929 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.178901    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"]
Oct 30 01:40:13.240930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.240882    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17087ce5-5b2b-4214-83c7-9f1fa5e5f769-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-vrmjk\" (UID: \"17087ce5-5b2b-4214-83c7-9f1fa5e5f769\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.240930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.240936    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mb9h\" (UniqueName: \"kubernetes.io/projected/17087ce5-5b2b-4214-83c7-9f1fa5e5f769-kube-api-access-6mb9h\") pod \"kuadrant-operator-controller-manager-57bb9bfff-vrmjk\" (UID: \"17087ce5-5b2b-4214-83c7-9f1fa5e5f769\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.342333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.342240    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17087ce5-5b2b-4214-83c7-9f1fa5e5f769-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-vrmjk\" (UID: \"17087ce5-5b2b-4214-83c7-9f1fa5e5f769\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.342333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.342297    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mb9h\" (UniqueName: \"kubernetes.io/projected/17087ce5-5b2b-4214-83c7-9f1fa5e5f769-kube-api-access-6mb9h\") pod \"kuadrant-operator-controller-manager-57bb9bfff-vrmjk\" (UID: \"17087ce5-5b2b-4214-83c7-9f1fa5e5f769\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.342776 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.342636    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17087ce5-5b2b-4214-83c7-9f1fa5e5f769-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-vrmjk\" (UID: \"17087ce5-5b2b-4214-83c7-9f1fa5e5f769\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.364996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.364961    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mb9h\" (UniqueName: \"kubernetes.io/projected/17087ce5-5b2b-4214-83c7-9f1fa5e5f769-kube-api-access-6mb9h\") pod \"kuadrant-operator-controller-manager-57bb9bfff-vrmjk\" (UID: \"17087ce5-5b2b-4214-83c7-9f1fa5e5f769\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.461934 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.461901    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:13.782080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:13.781889    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"]
Oct 30 01:40:13.784819 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:40:13.784787    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17087ce5_5b2b_4214_83c7_9f1fa5e5f769.slice/crio-4d15bcf8139d30ae9ea7cf29562da93bafbea8b9f822dc7df61872b86e5ac183 WatchSource:0}: Error finding container 4d15bcf8139d30ae9ea7cf29562da93bafbea8b9f822dc7df61872b86e5ac183: Status 404 returned error can't find the container with id 4d15bcf8139d30ae9ea7cf29562da93bafbea8b9f822dc7df61872b86e5ac183
Oct 30 01:40:14.577534 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:14.577496    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b869940a-1443-420e-ae8e-577c66d5e875" path="/var/lib/kubelet/pods/b869940a-1443-420e-ae8e-577c66d5e875/volumes"
Oct 30 01:40:14.750950 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:14.750911    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk" event={"ID":"17087ce5-5b2b-4214-83c7-9f1fa5e5f769","Type":"ContainerStarted","Data":"8f1569bb507ca6bc0e84b14e3ecce35764885e1d43299c09d56b3622e780eae0"}
Oct 30 01:40:14.751168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:14.751075    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:14.751168 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:14.751106    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk" event={"ID":"17087ce5-5b2b-4214-83c7-9f1fa5e5f769","Type":"ContainerStarted","Data":"4d15bcf8139d30ae9ea7cf29562da93bafbea8b9f822dc7df61872b86e5ac183"}
Oct 30 01:40:14.813013 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:14.812958    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk" podStartSLOduration=1.812944115 podStartE2EDuration="1.812944115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:14.801219582 +0000 UTC m=+3456.348910078" watchObservedRunningTime="2026-10-30 01:40:14.812944115 +0000 UTC m=+3456.360634611"
Oct 30 01:40:25.759245 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:25.759211    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-vrmjk"
Oct 30 01:40:34.028605 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:34.028556    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"]
Oct 30 01:40:34.038755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:34.038710    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxj6j"]
Oct 30 01:40:34.338125 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:34.338023    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:40:34.577829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:34.577786    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4172dbec-b8da-4384-9546-90e199daf07f" path="/var/lib/kubelet/pods/4172dbec-b8da-4384-9546-90e199daf07f/volumes"
Oct 30 01:40:38.473553 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.473498    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"]
Oct 30 01:40:38.478867 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.478842    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.496313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.496280    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"]
Oct 30 01:40:38.505522 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.505491    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:40:38.505893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.505565    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:40:38.505893 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.505571    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:40:38.506110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.505711    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:40:38.527351 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.527301    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzqj\" (UniqueName: \"kubernetes.io/projected/2bbddb32-a01f-49e1-9c10-68009c76f775-kube-api-access-rqzqj\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.527602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.527391    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.527602 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.527537    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.628163 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.628123    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzqj\" (UniqueName: \"kubernetes.io/projected/2bbddb32-a01f-49e1-9c10-68009c76f775-kube-api-access-rqzqj\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.628440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.628210    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.628440 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.628268    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.629038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.629010    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.630817 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.630788    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.646936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.646902    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzqj\" (UniqueName: \"kubernetes.io/projected/2bbddb32-a01f-49e1-9c10-68009c76f775-kube-api-access-rqzqj\") pod \"maas-api-revoke-sub-d576e68b-46dtl\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:38.806708 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.806622    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:40:38.813707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:38.813678    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:39.104196 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:40:39.104110    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:40:39.114109 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:39.114080    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"]
Oct 30 01:40:39.116333 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:40:39.116300    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbddb32_a01f_49e1_9c10_68009c76f775.slice/crio-694f030f5a78c9731c209c1d2ce6ef6dad737f249f6e24caf826fb2b7b31b936 WatchSource:0}: Error finding container 694f030f5a78c9731c209c1d2ce6ef6dad737f249f6e24caf826fb2b7b31b936: Status 404 returned error can't find the container with id 694f030f5a78c9731c209c1d2ce6ef6dad737f249f6e24caf826fb2b7b31b936
Oct 30 01:40:40.031390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:40.031352    5104 generic.go:356] "Generic (PLEG): container finished" podID="2bbddb32-a01f-49e1-9c10-68009c76f775" containerID="d793f31f7a5f706cecd2adf52c7870e4fd5f559b5bfffe083b062392ba1dc56a" exitCode=0
Oct 30 01:40:40.031847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:40.031411    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl" event={"ID":"2bbddb32-a01f-49e1-9c10-68009c76f775","Type":"ContainerDied","Data":"d793f31f7a5f706cecd2adf52c7870e4fd5f559b5bfffe083b062392ba1dc56a"}
Oct 30 01:40:40.031847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:40.031486    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl" event={"ID":"2bbddb32-a01f-49e1-9c10-68009c76f775","Type":"ContainerStarted","Data":"694f030f5a78c9731c209c1d2ce6ef6dad737f249f6e24caf826fb2b7b31b936"}
Oct 30 01:40:41.184112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.184083    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:41.254699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.254652    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-openshift-ca\") pod \"2bbddb32-a01f-49e1-9c10-68009c76f775\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") "
Oct 30 01:40:41.254921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.254803    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-xks-ca\") pod \"2bbddb32-a01f-49e1-9c10-68009c76f775\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") "
Oct 30 01:40:41.254921 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.254849    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbddb32-a01f-49e1-9c10-68009c76f775-kube-api-access-rqzqj\" (UniqueName: \"kubernetes.io/projected/2bbddb32-a01f-49e1-9c10-68009c76f775-kube-api-access-rqzqj\") pod \"2bbddb32-a01f-49e1-9c10-68009c76f775\" (UID: \"2bbddb32-a01f-49e1-9c10-68009c76f775\") "
Oct 30 01:40:41.255159 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.255128    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-openshift-ca" pod "2bbddb32-a01f-49e1-9c10-68009c76f775" (UID: "2bbddb32-a01f-49e1-9c10-68009c76f775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:40:41.255272 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.255253    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:40:41.257060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.257021    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-xks-ca" pod "2bbddb32-a01f-49e1-9c10-68009c76f775" (UID: "2bbddb32-a01f-49e1-9c10-68009c76f775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:40:41.257201 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.257131    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbddb32-a01f-49e1-9c10-68009c76f775-kube-api-access-rqzqj" pod "2bbddb32-a01f-49e1-9c10-68009c76f775" (UID: "2bbddb32-a01f-49e1-9c10-68009c76f775"). InnerVolumeSpecName "kube-api-access-rqzqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:41.356789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.356698    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbddb32-a01f-49e1-9c10-68009c76f775-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:40:41.356789 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:41.356735    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzqj\" (UniqueName: \"kubernetes.io/projected/2bbddb32-a01f-49e1-9c10-68009c76f775-kube-api-access-rqzqj\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:40:42.043349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:42.043318    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"
Oct 30 01:40:42.043560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:42.043314    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl" event={"ID":"2bbddb32-a01f-49e1-9c10-68009c76f775","Type":"ContainerDied","Data":"694f030f5a78c9731c209c1d2ce6ef6dad737f249f6e24caf826fb2b7b31b936"}
Oct 30 01:40:42.043560 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:42.043450    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="694f030f5a78c9731c209c1d2ce6ef6dad737f249f6e24caf826fb2b7b31b936"
Oct 30 01:40:49.615833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:49.615793    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:40:53.324479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.323523    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"]
Oct 30 01:40:53.324479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.324245    5104 cpu_manager.go:414] "containerMap: removing container" podUID="2bbddb32-a01f-49e1-9c10-68009c76f775" containerName="revoke-keys"
Oct 30 01:40:53.324479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.324267    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbddb32-a01f-49e1-9c10-68009c76f775" containerName="revoke-keys"
Oct 30 01:40:53.324479 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.324414    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbddb32-a01f-49e1-9c10-68009c76f775" containerName="revoke-keys"
Oct 30 01:40:53.338527 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.328666    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.379482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.349525    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"]
Oct 30 01:40:53.404480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.361382    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:40:53.408480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.361413    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:40:53.424699 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.361517    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:40:53.436481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.361598    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:40:53.548490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.540323    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.548809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.548782    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.549003 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.548988    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrr6\" (UniqueName: \"kubernetes.io/projected/010ebff9-37da-465f-ad44-49fb0fa2d6cd-kube-api-access-tvrr6\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.691656 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.691482    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.710483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.696376    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.717483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.716031    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrr6\" (UniqueName: \"kubernetes.io/projected/010ebff9-37da-465f-ad44-49fb0fa2d6cd-kube-api-access-tvrr6\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.717483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.716160    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.742485 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.735220    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.805483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.805315    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrr6\" (UniqueName: \"kubernetes.io/projected/010ebff9-37da-465f-ad44-49fb0fa2d6cd-kube-api-access-tvrr6\") pod \"maas-api-revoke-sub-d2bc93bc-f9vrr\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:53.854483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:53.853816    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:54.854674 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:54.854638    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"]
Oct 30 01:40:55.183483 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:55.182471    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr" event={"ID":"010ebff9-37da-465f-ad44-49fb0fa2d6cd","Type":"ContainerStarted","Data":"94946e16399e017fa9c2854a515f931264ce26d566c5bb7db4e7ed3577435517"}
Oct 30 01:40:56.029073 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.029030    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"]
Oct 30 01:40:56.040078 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.040041    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vf4dv"]
Oct 30 01:40:56.120238 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.120206    5104 scope.go:122] "RemoveContainer" containerID="6734574f00f14ab06a3abd5887881478bdb214734fc97b7e8ded2918d5f8e8d0"
Oct 30 01:40:56.132293 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.132263    5104 scope.go:122] "RemoveContainer" containerID="9299b2fc2faa2db12e9c0d2286b7658c01edeefa40531f6b4631a151633d6580"
Oct 30 01:40:56.141380 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.141354    5104 scope.go:122] "RemoveContainer" containerID="cb573bafa7e14eef98df02388a87c723ff2de867bfb472a3511ca31096fff7bf"
Oct 30 01:40:56.189603 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.189572    5104 generic.go:356] "Generic (PLEG): container finished" podID="010ebff9-37da-465f-ad44-49fb0fa2d6cd" containerID="da6b3fe14919bd8aab277953842ce73bce685d5bc2cd611101c2a69f0197ae57" exitCode=0
Oct 30 01:40:56.189778 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.189658    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr" event={"ID":"010ebff9-37da-465f-ad44-49fb0fa2d6cd","Type":"ContainerDied","Data":"da6b3fe14919bd8aab277953842ce73bce685d5bc2cd611101c2a69f0197ae57"}
Oct 30 01:40:56.576183 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:56.576143    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27b7c46-e419-4dfa-bdf4-e17e3f1cb552" path="/var/lib/kubelet/pods/c27b7c46-e419-4dfa-bdf4-e17e3f1cb552/volumes"
Oct 30 01:40:57.338539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.338511    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:57.465569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.465416    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-openshift-ca\") pod \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") "
Oct 30 01:40:57.465569 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.465515    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-xks-ca\") pod \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") "
Oct 30 01:40:57.465840 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.465641    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010ebff9-37da-465f-ad44-49fb0fa2d6cd-kube-api-access-tvrr6\" (UniqueName: \"kubernetes.io/projected/010ebff9-37da-465f-ad44-49fb0fa2d6cd-kube-api-access-tvrr6\") pod \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\" (UID: \"010ebff9-37da-465f-ad44-49fb0fa2d6cd\") "
Oct 30 01:40:57.465911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.465877    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-openshift-ca" pod "010ebff9-37da-465f-ad44-49fb0fa2d6cd" (UID: "010ebff9-37da-465f-ad44-49fb0fa2d6cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:40:57.467882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.467848    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-xks-ca" pod "010ebff9-37da-465f-ad44-49fb0fa2d6cd" (UID: "010ebff9-37da-465f-ad44-49fb0fa2d6cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:40:57.468015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.467925    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010ebff9-37da-465f-ad44-49fb0fa2d6cd-kube-api-access-tvrr6" pod "010ebff9-37da-465f-ad44-49fb0fa2d6cd" (UID: "010ebff9-37da-465f-ad44-49fb0fa2d6cd"). InnerVolumeSpecName "kube-api-access-tvrr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:57.566393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.566356    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:40:57.566393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.566386    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrr6\" (UniqueName: \"kubernetes.io/projected/010ebff9-37da-465f-ad44-49fb0fa2d6cd-kube-api-access-tvrr6\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:40:57.566393 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:57.566398    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ebff9-37da-465f-ad44-49fb0fa2d6cd-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:40:58.201768 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:58.201733    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"
Oct 30 01:40:58.201958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:58.201769    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr" event={"ID":"010ebff9-37da-465f-ad44-49fb0fa2d6cd","Type":"ContainerDied","Data":"94946e16399e017fa9c2854a515f931264ce26d566c5bb7db4e7ed3577435517"}
Oct 30 01:40:58.201958 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:40:58.201821    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94946e16399e017fa9c2854a515f931264ce26d566c5bb7db4e7ed3577435517"
Oct 30 01:41:09.103836 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:41:09.103800    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:41:10.688833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.688792    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"]
Oct 30 01:41:10.689542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.689504    5104 cpu_manager.go:414] "containerMap: removing container" podUID="010ebff9-37da-465f-ad44-49fb0fa2d6cd" containerName="revoke-keys"
Oct 30 01:41:10.689542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.689533    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010ebff9-37da-465f-ad44-49fb0fa2d6cd" containerName="revoke-keys"
Oct 30 01:41:10.689724 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.689688    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="010ebff9-37da-465f-ad44-49fb0fa2d6cd" containerName="revoke-keys"
Oct 30 01:41:10.693662 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.693616    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.724882 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.724837    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:41:10.725131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.725101    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nss4l\""
Oct 30 01:41:10.725256 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.725118    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b63f8a-serving-cert\""
Oct 30 01:41:10.725409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.725277    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:41:10.725409 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.725339    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:41:10.728212 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.728174    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"]
Oct 30 01:41:10.793701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.793653    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.793885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.793783    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b812a-2d91-4d91-a584-ff891d0c04d1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.793885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.793817    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.793885 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.793872    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txxs\" (UniqueName: \"kubernetes.io/projected/448b812a-2d91-4d91-a584-ff891d0c04d1-kube-api-access-2txxs\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.894889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.894854    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b812a-2d91-4d91-a584-ff891d0c04d1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.894889 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.894893    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.895193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.894941    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txxs\" (UniqueName: \"kubernetes.io/projected/448b812a-2d91-4d91-a584-ff891d0c04d1-kube-api-access-2txxs\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.895193 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.895012    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.895822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.895789    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b812a-2d91-4d91-a584-ff891d0c04d1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.897911 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.897879    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.898051 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.897945    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:10.914546 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:10.914510    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txxs\" (UniqueName: \"kubernetes.io/projected/448b812a-2d91-4d91-a584-ff891d0c04d1-kube-api-access-2txxs\") pod \"maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:11.008364 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.008272    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:11.067196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.066203    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"]
Oct 30 01:41:11.071533 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.071511    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.085991 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.085958    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-640f0b93-dockercfg-4rhqd\""
Oct 30 01:41:11.087390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.087360    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b63f8a\""
Oct 30 01:41:11.091833 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.089821    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"]
Oct 30 01:41:11.097944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.097909    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nfz\" (UniqueName: \"kubernetes.io/projected/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-kube-api-access-c4nfz\") pod \"payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.098098 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.098023    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.199249 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.199210    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nfz\" (UniqueName: \"kubernetes.io/projected/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-kube-api-access-c4nfz\") pod \"payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.199500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.199292    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.200435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.200081    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.218604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.218563    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nfz\" (UniqueName: \"kubernetes.io/projected/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-kube-api-access-c4nfz\") pod \"payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.311763 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.311728    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"]
Oct 30 01:41:11.314643 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:41:11.314600    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448b812a_2d91_4d91_a584_ff891d0c04d1.slice/crio-52c68aa9a84cb7f3761755aad1883221d64ae51ffee9b2ebb3855985f09c38de WatchSource:0}: Error finding container 52c68aa9a84cb7f3761755aad1883221d64ae51ffee9b2ebb3855985f09c38de: Status 404 returned error can't find the container with id 52c68aa9a84cb7f3761755aad1883221d64ae51ffee9b2ebb3855985f09c38de
Oct 30 01:41:11.316322 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.316299    5104 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:41:11.395990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.395955    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:11.691416 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:11.691374    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"]
Oct 30 01:41:11.694240 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:41:11.694205    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce7a95e_3358_4f2e_aa5d_4db572e3a404.slice/crio-70907d4da5461be1a81c923c8d7864247dd17959161c22502ca8bcebfb3f0ea0 WatchSource:0}: Error finding container 70907d4da5461be1a81c923c8d7864247dd17959161c22502ca8bcebfb3f0ea0: Status 404 returned error can't find the container with id 70907d4da5461be1a81c923c8d7864247dd17959161c22502ca8bcebfb3f0ea0
Oct 30 01:41:12.285142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.285098    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" event={"ID":"0ce7a95e-3358-4f2e-aa5d-4db572e3a404","Type":"ContainerStarted","Data":"c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc"}
Oct 30 01:41:12.285372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.285170    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" event={"ID":"0ce7a95e-3358-4f2e-aa5d-4db572e3a404","Type":"ContainerStarted","Data":"70907d4da5461be1a81c923c8d7864247dd17959161c22502ca8bcebfb3f0ea0"}
Oct 30 01:41:12.285372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.285296    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:12.287305 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.287278    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw" event={"ID":"448b812a-2d91-4d91-a584-ff891d0c04d1","Type":"ContainerStarted","Data":"fbb2de1f08d186cc0d98acd66d652af9b8dd904f9fb2b5dcb9a6c112fa43e9de"}
Oct 30 01:41:12.287486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.287322    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw" event={"ID":"448b812a-2d91-4d91-a584-ff891d0c04d1","Type":"ContainerStarted","Data":"52c68aa9a84cb7f3761755aad1883221d64ae51ffee9b2ebb3855985f09c38de"}
Oct 30 01:41:12.287486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.287381    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:12.321946 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.321889    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" podStartSLOduration=1.321876714 podStartE2EDuration="1.321876714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:12.321446565 +0000 UTC m=+3513.869137062" watchObservedRunningTime="2026-10-30 01:41:12.321876714 +0000 UTC m=+3513.869567207"
Oct 30 01:41:12.361804 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:12.361734    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw" podStartSLOduration=1.983813753 podStartE2EDuration="2.36171636s" totalImagesPullingTime="377.902607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:12.359647861 +0000 UTC m=+3513.907338356" watchObservedRunningTime="2026-10-30 01:41:12.36171636 +0000 UTC m=+3513.909406856"
Oct 30 01:41:14.759628 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:14.755340    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:41:16.026972 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:16.026932    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"]
Oct 30 01:41:16.038110 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:16.038073    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-c9r97"]
Oct 30 01:41:16.576401 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:16.576364    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f415eead-8f39-4961-8d9d-a2ab819662d5" path="/var/lib/kubelet/pods/f415eead-8f39-4961-8d9d-a2ab819662d5/volumes"
Oct 30 01:41:18.299971 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:18.299942    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:41:23.296774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:23.296737    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:41:29.328821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:29.328779    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:41:37.027255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:37.027218    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"]
Oct 30 01:41:37.038711 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:37.038681    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pfjnr"]
Oct 30 01:41:38.576731 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:38.576699    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c8455e-dd34-44d5-bab0-04ed95ab12bf" path="/var/lib/kubelet/pods/59c8455e-dd34-44d5-bab0-04ed95ab12bf/volumes"
Oct 30 01:41:39.104746 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:41:39.104706    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:41:43.453472 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:43.453417    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:41:56.173368 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:56.173335    5104 scope.go:122] "RemoveContainer" containerID="1732e6553bad0e799d7b0de602bf2807bf3fb741a707dd95fc1f360f41d9622d"
Oct 30 01:41:56.182719 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:56.182694    5104 scope.go:122] "RemoveContainer" containerID="3d03e28ba104f86c0c4db3ed6666191a13818e0614ad30f8e51beb84bfe9774d"
Oct 30 01:41:56.191872 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:56.191847    5104 scope.go:122] "RemoveContainer" containerID="e135ebbaaae1c6e137f64143ece70dff2c4a3148fac7ee27b609ff895fd09255"
Oct 30 01:41:56.201096 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:56.201073    5104 scope.go:122] "RemoveContainer" containerID="0a3995cb6b4e6b641d32020b001f9855e29d889ed29984eaaf3d55c298a3f02e"
Oct 30 01:41:56.845195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:41:56.845151    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:42:07.014806 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:07.014767    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"]
Oct 30 01:42:07.463565 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:07.463524    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"]
Oct 30 01:42:07.463871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:07.463841    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw" podUID="448b812a-2d91-4d91-a584-ff891d0c04d1" containerName="maas-api" containerID="cri-o://fbb2de1f08d186cc0d98acd66d652af9b8dd904f9fb2b5dcb9a6c112fa43e9de" gracePeriod=30
Oct 30 01:42:08.295152 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.294417    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw" podUID="448b812a-2d91-4d91-a584-ff891d0c04d1" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.197:8443/health\": dial tcp 10.132.1.197:8443: connect: connection refused"
Oct 30 01:42:08.641382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.640959    5104 generic.go:356] "Generic (PLEG): container finished" podID="448b812a-2d91-4d91-a584-ff891d0c04d1" containerID="fbb2de1f08d186cc0d98acd66d652af9b8dd904f9fb2b5dcb9a6c112fa43e9de" exitCode=0
Oct 30 01:42:08.641626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.641384    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw" event={"ID":"448b812a-2d91-4d91-a584-ff891d0c04d1","Type":"ContainerDied","Data":"fbb2de1f08d186cc0d98acd66d652af9b8dd904f9fb2b5dcb9a6c112fa43e9de"}
Oct 30 01:42:08.728174 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.728144    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:42:08.840883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.840791    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-maas-api-tls\") pod \"448b812a-2d91-4d91-a584-ff891d0c04d1\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") "
Oct 30 01:42:08.840883 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.840875    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448b812a-2d91-4d91-a584-ff891d0c04d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b812a-2d91-4d91-a584-ff891d0c04d1-openshift-service-ca\") pod \"448b812a-2d91-4d91-a584-ff891d0c04d1\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") "
Oct 30 01:42:08.841160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.840973    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-metrics-certs\") pod \"448b812a-2d91-4d91-a584-ff891d0c04d1\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") "
Oct 30 01:42:08.841160 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.841105    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448b812a-2d91-4d91-a584-ff891d0c04d1-kube-api-access-2txxs\" (UniqueName: \"kubernetes.io/projected/448b812a-2d91-4d91-a584-ff891d0c04d1-kube-api-access-2txxs\") pod \"448b812a-2d91-4d91-a584-ff891d0c04d1\" (UID: \"448b812a-2d91-4d91-a584-ff891d0c04d1\") "
Oct 30 01:42:08.841325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.841288    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448b812a-2d91-4d91-a584-ff891d0c04d1-openshift-service-ca" pod "448b812a-2d91-4d91-a584-ff891d0c04d1" (UID: "448b812a-2d91-4d91-a584-ff891d0c04d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:08.841447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.841428    5104 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/448b812a-2d91-4d91-a584-ff891d0c04d1-openshift-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:42:08.843307 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.843276    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448b812a-2d91-4d91-a584-ff891d0c04d1-kube-api-access-2txxs" pod "448b812a-2d91-4d91-a584-ff891d0c04d1" (UID: "448b812a-2d91-4d91-a584-ff891d0c04d1"). InnerVolumeSpecName "kube-api-access-2txxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:08.843448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.843287    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-metrics-certs" pod "448b812a-2d91-4d91-a584-ff891d0c04d1" (UID: "448b812a-2d91-4d91-a584-ff891d0c04d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:08.843448 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.843378    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-maas-api-tls" pod "448b812a-2d91-4d91-a584-ff891d0c04d1" (UID: "448b812a-2d91-4d91-a584-ff891d0c04d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:08.942323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.942285    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:42:08.942323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.942316    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txxs\" (UniqueName: \"kubernetes.io/projected/448b812a-2d91-4d91-a584-ff891d0c04d1-kube-api-access-2txxs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:42:08.942323 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:08.942327    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/448b812a-2d91-4d91-a584-ff891d0c04d1-maas-api-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:42:09.104933 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:42:09.104845    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:42:09.648390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:09.648351    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"
Oct 30 01:42:09.648390 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:09.648372    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw" event={"ID":"448b812a-2d91-4d91-a584-ff891d0c04d1","Type":"ContainerDied","Data":"52c68aa9a84cb7f3761755aad1883221d64ae51ffee9b2ebb3855985f09c38de"}
Oct 30 01:42:09.648961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:09.648441    5104 scope.go:122] "RemoveContainer" containerID="fbb2de1f08d186cc0d98acd66d652af9b8dd904f9fb2b5dcb9a6c112fa43e9de"
Oct 30 01:42:09.698296 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:09.698253    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"]
Oct 30 01:42:09.710425 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:09.710390    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b63f8a-84cc686d55-r9qcw"]
Oct 30 01:42:09.852613 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:09.852555    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:42:10.577956 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:10.577913    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448b812a-2d91-4d91-a584-ff891d0c04d1" path="/var/lib/kubelet/pods/448b812a-2d91-4d91-a584-ff891d0c04d1/volumes"
Oct 30 01:42:12.015785 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:12.015741    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" podUID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" containerName="payload-pre-processing" containerID="cri-o://c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc" gracePeriod=25
Oct 30 01:42:13.294849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:13.294812    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" podUID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:22.340821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:22.340778    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:42:23.295357 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:23.295307    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" podUID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:26.495659 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:26.495612    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:42:26.496187 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:26.495710    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:42:32.281870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.281800    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:42:32.333961 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.333924    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-plugins-config-volume\") pod \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") "
Oct 30 01:42:32.334145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.334038    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-kube-api-access-c4nfz\" (UniqueName: \"kubernetes.io/projected/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-kube-api-access-c4nfz\") pod \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\" (UID: \"0ce7a95e-3358-4f2e-aa5d-4db572e3a404\") "
Oct 30 01:42:32.334343 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.334318    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-plugins-config-volume" pod "0ce7a95e-3358-4f2e-aa5d-4db572e3a404" (UID: "0ce7a95e-3358-4f2e-aa5d-4db572e3a404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:32.334436 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.334414    5104 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-plugins-config-volume\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:42:32.336300 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.336275    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-kube-api-access-c4nfz" pod "0ce7a95e-3358-4f2e-aa5d-4db572e3a404" (UID: "0ce7a95e-3358-4f2e-aa5d-4db572e3a404"). InnerVolumeSpecName "kube-api-access-c4nfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:32.435809 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.435772    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4nfz\" (UniqueName: \"kubernetes.io/projected/0ce7a95e-3358-4f2e-aa5d-4db572e3a404-kube-api-access-c4nfz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:42:32.821435 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.821399    5104 generic.go:356] "Generic (PLEG): container finished" podID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" containerID="c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc" exitCode=0
Oct 30 01:42:32.821683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.821449    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" event={"ID":"0ce7a95e-3358-4f2e-aa5d-4db572e3a404","Type":"ContainerDied","Data":"c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc"}
Oct 30 01:42:32.821683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.821473    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"
Oct 30 01:42:32.821683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.821508    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb" event={"ID":"0ce7a95e-3358-4f2e-aa5d-4db572e3a404","Type":"ContainerDied","Data":"70907d4da5461be1a81c923c8d7864247dd17959161c22502ca8bcebfb3f0ea0"}
Oct 30 01:42:32.821683 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.821531    5104 scope.go:122] "RemoveContainer" containerID="c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc"
Oct 30 01:42:32.831133 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.831104    5104 scope.go:122] "RemoveContainer" containerID="c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc"
Oct 30 01:42:32.831412 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:42:32.831388    5104 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc\": container with ID starting with c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc not found: ID does not exist" containerID="c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc"
Oct 30 01:42:32.831490 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.831422    5104 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc"} err="failed to get container status \"c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc\": rpc error: code = NotFound desc = could not find container \"c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc\": container with ID starting with c634d1c0ee24ac0b34d1957d31450ab4831af0d16f24767b0d9026f1c5e535cc not found: ID does not exist"
Oct 30 01:42:32.870424 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.870330    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"]
Oct 30 01:42:32.881670 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:32.881636    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b63f8a-9874c5bmrb"]
Oct 30 01:42:34.576252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:34.576213    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" path="/var/lib/kubelet/pods/0ce7a95e-3358-4f2e-aa5d-4db572e3a404/volumes"
Oct 30 01:42:39.111368 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:42:39.111329    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:42:39.783430 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:39.783392    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:42:39.792080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:39.792042    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:42:39.795997 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:39.795975    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:42:39.803604 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:39.803580    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tntr4_085c11e9-3ae8-4a1d-abe7-edc7dcad9804/ovn-acl-logging/0.log"
Oct 30 01:42:41.436313 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:41.436274    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:42:55.133952 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:55.133900    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:42:56.243434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.243396    5104 scope.go:122] "RemoveContainer" containerID="778bc6d0ad43c43bdb31287d037074736d3ec66dad1a137b6fc363c1ede7ba28"
Oct 30 01:42:56.259671 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.259641    5104 scope.go:122] "RemoveContainer" containerID="44ffa4d71b7634413d092291ff083f6ef5212e77d2e8b0011153bde58f599e64"
Oct 30 01:42:56.270316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.270291    5104 scope.go:122] "RemoveContainer" containerID="64c11260bc70873618c5e9830066485f3db4318c2b17622cde781ef7e2c542d3"
Oct 30 01:42:56.302258 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.302182    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"]
Oct 30 01:42:56.302936 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.302910    5104 cpu_manager.go:414] "containerMap: removing container" podUID="448b812a-2d91-4d91-a584-ff891d0c04d1" containerName="maas-api"
Oct 30 01:42:56.303045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.302938    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448b812a-2d91-4d91-a584-ff891d0c04d1" containerName="maas-api"
Oct 30 01:42:56.303045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.302966    5104 cpu_manager.go:414] "containerMap: removing container" podUID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" containerName="payload-pre-processing"
Oct 30 01:42:56.303045 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.302975    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" containerName="payload-pre-processing"
Oct 30 01:42:56.303236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.303111    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="448b812a-2d91-4d91-a584-ff891d0c04d1" containerName="maas-api"
Oct 30 01:42:56.303236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.303126    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce7a95e-3358-4f2e-aa5d-4db572e3a404" containerName="payload-pre-processing"
Oct 30 01:42:56.317099 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.313430    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.328782 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.328661    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:42:56.342095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.337319    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:42:56.342095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.341406    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:42:56.342095 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.341524    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:42:56.344113 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.344029    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"]
Oct 30 01:42:56.499732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.499609    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdd29\" (UniqueName: \"kubernetes.io/projected/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-kube-api-access-pdd29\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.499732 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.499686    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.500015 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.499839    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.600930 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.600870    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdd29\" (UniqueName: \"kubernetes.io/projected/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-kube-api-access-pdd29\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.601167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.600944    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.601167 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.601003    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.601964 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.601928    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.605317 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.605278    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.625532 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.625420    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdd29\" (UniqueName: \"kubernetes.io/projected/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-kube-api-access-pdd29\") pod \"maas-api-revoke-sub-452b6d7b-bpfbx\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:56.663093 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:56.663045    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:42:57.025672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:57.013667    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"]
Oct 30 01:42:57.057777 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:57.057685    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx" event={"ID":"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6","Type":"ContainerStarted","Data":"0c14e9e347f01bf0123f4693538636df5d3ff2dc0b8e6d43fe949e51f7d1f65a"}
Oct 30 01:42:58.072487 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:58.071783    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx" event={"ID":"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6","Type":"ContainerStarted","Data":"595cbc52e3829e0aebc9a01f8bb129732b30003461ff78257faa3906edef3efd"}
Oct 30 01:42:58.699481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:58.698557    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:42:58.699481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:58.698651    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:42:59.157235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:59.156560    5104 generic.go:356] "Generic (PLEG): container finished" podID="7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6" containerID="595cbc52e3829e0aebc9a01f8bb129732b30003461ff78257faa3906edef3efd" exitCode=0
Oct 30 01:42:59.157235 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:42:59.156736    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx" event={"ID":"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6","Type":"ContainerDied","Data":"595cbc52e3829e0aebc9a01f8bb129732b30003461ff78257faa3906edef3efd"}
Oct 30 01:43:00.298413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.298385    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:43:00.356173 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.356134    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-xks-ca\") pod \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") "
Oct 30 01:43:00.356372 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.356333    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-openshift-ca\") pod \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") "
Oct 30 01:43:00.356432 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.356391    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-kube-api-access-pdd29\" (UniqueName: \"kubernetes.io/projected/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-kube-api-access-pdd29\") pod \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\" (UID: \"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6\") "
Oct 30 01:43:00.356784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.356751    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-openshift-ca" pod "7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6" (UID: "7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:00.358420 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.358393    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-xks-ca" pod "7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6" (UID: "7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:00.358900 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.358876    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-kube-api-access-pdd29" pod "7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6" (UID: "7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6"). InnerVolumeSpecName "kube-api-access-pdd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:00.457898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.457865    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:00.457898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.457893    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdd29\" (UniqueName: \"kubernetes.io/projected/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-kube-api-access-pdd29\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:00.457898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:00.457903    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:01.167619 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:01.167578    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx" event={"ID":"7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6","Type":"ContainerDied","Data":"0c14e9e347f01bf0123f4693538636df5d3ff2dc0b8e6d43fe949e51f7d1f65a"}
Oct 30 01:43:01.167870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:01.167646    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c14e9e347f01bf0123f4693538636df5d3ff2dc0b8e6d43fe949e51f7d1f65a"
Oct 30 01:43:01.167870 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:01.167597    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bpfbx"
Oct 30 01:43:07.721774 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:07.721735    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:43:09.117738 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:43:09.117668    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:43:11.509626 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:11.509576    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pc85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:43:11.510285 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:11.510247    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pc85x" podUID="9a10bb9b-ade0-4b26-9b51-8133ffad51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:43:13.697481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:13.696569    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwnrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:43:13.697481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:13.696662    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwnrr" podUID="042bc9a4-74e8-4187-ac04-6afbf483756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:43:15.009697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.009655    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"]
Oct 30 01:43:15.010672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.010636    5104 cpu_manager.go:414] "containerMap: removing container" podUID="7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6" containerName="revoke-keys"
Oct 30 01:43:15.010672 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.010665    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6" containerName="revoke-keys"
Oct 30 01:43:15.010851 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.010770    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6" containerName="revoke-keys"
Oct 30 01:43:15.014043 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.014019    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.040408 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.040373    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:43:15.040704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.040430    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:43:15.040704 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.040429    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:43:15.040906 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.040508    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:43:15.043081 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.043050    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"]
Oct 30 01:43:15.077142 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.077099    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.077325 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.077272    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqcz\" (UniqueName: \"kubernetes.io/projected/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-kube-api-access-dfqcz\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.077387 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.077368    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.178520 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.178449    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.178716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.178558    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqcz\" (UniqueName: \"kubernetes.io/projected/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-kube-api-access-dfqcz\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.178716 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.178595    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.179276 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.179246    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.181289 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.181260    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.199054 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.199021    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqcz\" (UniqueName: \"kubernetes.io/projected/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-kube-api-access-dfqcz\") pod \"maas-api-revoke-sub-3e31a506-52kf6\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.329379 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.329278    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:15.441001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.440953    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:43:15.626631 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:15.626522    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"]
Oct 30 01:43:15.635001 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:43:15.634967    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d7c799_8744_4a37_a35b_acae0ebe5a1d.slice/crio-edfe6847586a00e8f286c5c0157fe7f9f0819213a6e72ba578e9b300b8793a26 WatchSource:0}: Error finding container edfe6847586a00e8f286c5c0157fe7f9f0819213a6e72ba578e9b300b8793a26: Status 404 returned error can't find the container with id edfe6847586a00e8f286c5c0157fe7f9f0819213a6e72ba578e9b300b8793a26
Oct 30 01:43:16.368116 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:16.368060    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6" event={"ID":"a3d7c799-8744-4a37-a35b-acae0ebe5a1d","Type":"ContainerStarted","Data":"fcca24f13ad9efc7bd224ecfe439510de9dcc452b49fa982515c217058ad0191"}
Oct 30 01:43:16.368831 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:16.368797    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6" event={"ID":"a3d7c799-8744-4a37-a35b-acae0ebe5a1d","Type":"ContainerStarted","Data":"edfe6847586a00e8f286c5c0157fe7f9f0819213a6e72ba578e9b300b8793a26"}
Oct 30 01:43:16.406944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:16.406808    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6" podStartSLOduration=2.406784427 podStartE2EDuration="2.406784427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:16.405146005 +0000 UTC m=+3637.952836523" watchObservedRunningTime="2026-10-30 01:43:16.406784427 +0000 UTC m=+3637.954474924"
Oct 30 01:43:17.408413 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:17.408372    5104 generic.go:356] "Generic (PLEG): container finished" podID="a3d7c799-8744-4a37-a35b-acae0ebe5a1d" containerID="fcca24f13ad9efc7bd224ecfe439510de9dcc452b49fa982515c217058ad0191" exitCode=0
Oct 30 01:43:17.409190 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:17.409157    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6" event={"ID":"a3d7c799-8744-4a37-a35b-acae0ebe5a1d","Type":"ContainerDied","Data":"fcca24f13ad9efc7bd224ecfe439510de9dcc452b49fa982515c217058ad0191"}
Oct 30 01:43:19.030049 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.030018    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:19.171495 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.171443    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-kube-api-access-dfqcz\" (UniqueName: \"kubernetes.io/projected/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-kube-api-access-dfqcz\") pod \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") "
Oct 30 01:43:19.171728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.171571    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-openshift-ca\") pod \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") "
Oct 30 01:43:19.171728 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.171634    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-xks-ca\") pod \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\" (UID: \"a3d7c799-8744-4a37-a35b-acae0ebe5a1d\") "
Oct 30 01:43:19.172058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.172024    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-openshift-ca" pod "a3d7c799-8744-4a37-a35b-acae0ebe5a1d" (UID: "a3d7c799-8744-4a37-a35b-acae0ebe5a1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:19.173832 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.173799    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-kube-api-access-dfqcz" pod "a3d7c799-8744-4a37-a35b-acae0ebe5a1d" (UID: "a3d7c799-8744-4a37-a35b-acae0ebe5a1d"). InnerVolumeSpecName "kube-api-access-dfqcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:19.173949 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.173831    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-xks-ca" pod "a3d7c799-8744-4a37-a35b-acae0ebe5a1d" (UID: "a3d7c799-8744-4a37-a35b-acae0ebe5a1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:19.273578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.273435    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:19.273578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.273524    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:19.273578 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.273540    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqcz\" (UniqueName: \"kubernetes.io/projected/a3d7c799-8744-4a37-a35b-acae0ebe5a1d-kube-api-access-dfqcz\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:19.451552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.451514    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6"
Oct 30 01:43:19.451762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.451517    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-52kf6" event={"ID":"a3d7c799-8744-4a37-a35b-acae0ebe5a1d","Type":"ContainerDied","Data":"edfe6847586a00e8f286c5c0157fe7f9f0819213a6e72ba578e9b300b8793a26"}
Oct 30 01:43:19.451762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:19.451653    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edfe6847586a00e8f286c5c0157fe7f9f0819213a6e72ba578e9b300b8793a26"
Oct 30 01:43:26.213421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:26.213376    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:43:30.340486 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:30.340434    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:43:37.355664 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.355621    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"]
Oct 30 01:43:37.356320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.356297    5104 cpu_manager.go:414] "containerMap: removing container" podUID="a3d7c799-8744-4a37-a35b-acae0ebe5a1d" containerName="revoke-keys"
Oct 30 01:43:37.356320 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.356320    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d7c799-8744-4a37-a35b-acae0ebe5a1d" containerName="revoke-keys"
Oct 30 01:43:37.356508 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.356397    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d7c799-8744-4a37-a35b-acae0ebe5a1d" containerName="revoke-keys"
Oct 30 01:43:37.360710 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.360691    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.379793 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.379769    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:43:37.381609 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.381579    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.381770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.381617    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.381770 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.381738    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnwm\" (UniqueName: \"kubernetes.io/projected/01d4a868-89d0-4dec-bf0f-5c49e0275793-kube-api-access-zlnwm\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.391358 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.391328    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:43:37.391591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.391353    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:43:37.391591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.391487    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\""
Oct 30 01:43:37.395155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.395128    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"]
Oct 30 01:43:37.482447 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.482410    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnwm\" (UniqueName: \"kubernetes.io/projected/01d4a868-89d0-4dec-bf0f-5c49e0275793-kube-api-access-zlnwm\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.482691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.482560    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.482691 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.482586    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.483263 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.483235    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.485083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.485060    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.501707 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.501677    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnwm\" (UniqueName: \"kubernetes.io/projected/01d4a868-89d0-4dec-bf0f-5c49e0275793-kube-api-access-zlnwm\") pod \"maas-api-revoke-sub-e2415be5-zbzs4\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.674060 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.674027    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:37.968743 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:37.968709    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"]
Oct 30 01:43:37.971137 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:43:37.971108    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d4a868_89d0_4dec_bf0f_5c49e0275793.slice/crio-e5c4c7be9bda7a483775380fe19ecda13a11513b5fc02bba2027f799aa4b1fcc WatchSource:0}: Error finding container e5c4c7be9bda7a483775380fe19ecda13a11513b5fc02bba2027f799aa4b1fcc: Status 404 returned error can't find the container with id e5c4c7be9bda7a483775380fe19ecda13a11513b5fc02bba2027f799aa4b1fcc
Oct 30 01:43:38.577355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:38.577265    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4" event={"ID":"01d4a868-89d0-4dec-bf0f-5c49e0275793","Type":"ContainerStarted","Data":"d8002a65992d47ea80f924e2b725f24773ee27e7a90ef667aa2b496067dd17de"}
Oct 30 01:43:38.577355 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:38.577325    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4" event={"ID":"01d4a868-89d0-4dec-bf0f-5c49e0275793","Type":"ContainerStarted","Data":"e5c4c7be9bda7a483775380fe19ecda13a11513b5fc02bba2027f799aa4b1fcc"}
Oct 30 01:43:38.612591 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:38.612536    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4" podStartSLOduration=1.612520531 podStartE2EDuration="1.612520531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:38.61057342 +0000 UTC m=+3660.158263915" watchObservedRunningTime="2026-10-30 01:43:38.612520531 +0000 UTC m=+3660.160211027"
Oct 30 01:43:39.104567 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:43:39.104531    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:43:39.580255 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:39.580217    5104 generic.go:356] "Generic (PLEG): container finished" podID="01d4a868-89d0-4dec-bf0f-5c49e0275793" containerID="d8002a65992d47ea80f924e2b725f24773ee27e7a90ef667aa2b496067dd17de" exitCode=0
Oct 30 01:43:39.580727 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:39.580305    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4" event={"ID":"01d4a868-89d0-4dec-bf0f-5c49e0275793","Type":"ContainerDied","Data":"d8002a65992d47ea80f924e2b725f24773ee27e7a90ef667aa2b496067dd17de"}
Oct 30 01:43:40.727500 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.727470    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:40.817879 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.817851    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-openshift-ca\") pod \"01d4a868-89d0-4dec-bf0f-5c49e0275793\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") "
Oct 30 01:43:40.818080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.817914    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d4a868-89d0-4dec-bf0f-5c49e0275793-kube-api-access-zlnwm\" (UniqueName: \"kubernetes.io/projected/01d4a868-89d0-4dec-bf0f-5c49e0275793-kube-api-access-zlnwm\") pod \"01d4a868-89d0-4dec-bf0f-5c49e0275793\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") "
Oct 30 01:43:40.818080 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.818025    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-xks-ca\") pod \"01d4a868-89d0-4dec-bf0f-5c49e0275793\" (UID: \"01d4a868-89d0-4dec-bf0f-5c49e0275793\") "
Oct 30 01:43:40.818281 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.818242    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-openshift-ca" pod "01d4a868-89d0-4dec-bf0f-5c49e0275793" (UID: "01d4a868-89d0-4dec-bf0f-5c49e0275793"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:40.818423 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.818396    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-openshift-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:40.820154 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.820120    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-xks-ca" pod "01d4a868-89d0-4dec-bf0f-5c49e0275793" (UID: "01d4a868-89d0-4dec-bf0f-5c49e0275793"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:40.820271 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.820201    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d4a868-89d0-4dec-bf0f-5c49e0275793-kube-api-access-zlnwm" pod "01d4a868-89d0-4dec-bf0f-5c49e0275793" (UID: "01d4a868-89d0-4dec-bf0f-5c49e0275793"). InnerVolumeSpecName "kube-api-access-zlnwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:40.919331 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.919301    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnwm\" (UniqueName: \"kubernetes.io/projected/01d4a868-89d0-4dec-bf0f-5c49e0275793-kube-api-access-zlnwm\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:40.919537 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:40.919334    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d4a868-89d0-4dec-bf0f-5c49e0275793-maas-api-xks-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:43:41.591847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:41.591813    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4"
Oct 30 01:43:41.591847 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:41.591840    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2415be5-zbzs4" event={"ID":"01d4a868-89d0-4dec-bf0f-5c49e0275793","Type":"ContainerDied","Data":"e5c4c7be9bda7a483775380fe19ecda13a11513b5fc02bba2027f799aa4b1fcc"}
Oct 30 01:43:41.592083 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:41.591882    5104 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5c4c7be9bda7a483775380fe19ecda13a11513b5fc02bba2027f799aa4b1fcc"
Oct 30 01:43:48.613518 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:48.613480    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:43:49.508938 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.508898    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"]
Oct 30 01:43:49.510123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.510093    5104 cpu_manager.go:414] "containerMap: removing container" podUID="01d4a868-89d0-4dec-bf0f-5c49e0275793" containerName="revoke-keys"
Oct 30 01:43:49.510273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.510124    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d4a868-89d0-4dec-bf0f-5c49e0275793" containerName="revoke-keys"
Oct 30 01:43:49.510273 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.510233    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d4a868-89d0-4dec-bf0f-5c49e0275793" containerName="revoke-keys"
Oct 30 01:43:49.513915 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.513885    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.544973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.544935    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:43:49.544973 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.544964    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nss4l\""
Oct 30 01:43:49.545328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.545021    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:43:49.545328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.545033    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:43:49.545328 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.545135    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 01:43:49.548653 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.548619    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"]
Oct 30 01:43:49.597517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.597470    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59szn\" (UniqueName: \"kubernetes.io/projected/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-kube-api-access-59szn\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.597741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.597597    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-metrics-certs\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.597741 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.597700    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-openshift-service-ca\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.597890 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.597747    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-maas-api-tls\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.698905 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.698865    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-openshift-service-ca\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.699536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.698917    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-maas-api-tls\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.699536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.699002    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59szn\" (UniqueName: \"kubernetes.io/projected/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-kube-api-access-59szn\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.699536 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.699024    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-metrics-certs\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.699841 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.699812    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-openshift-service-ca\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.702208 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.702179    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-metrics-certs\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.702324 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.702190    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-maas-api-tls\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.719421 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.719387    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59szn\" (UniqueName: \"kubernetes.io/projected/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-kube-api-access-59szn\") pod \"maas-api-5c48877c75-r5dwf\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:49.830166 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:49.830082    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:50.141550 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:50.141432    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"]
Oct 30 01:43:50.148676 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:43:50.148519    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2e6fea_4b06_48c2_b00c_2e58f86a2870.slice/crio-1e0ec63201b27ef640296bff4af92a050405f68d5d25e910e6ffe1c044b4c239 WatchSource:0}: Error finding container 1e0ec63201b27ef640296bff4af92a050405f68d5d25e910e6ffe1c044b4c239: Status 404 returned error can't find the container with id 1e0ec63201b27ef640296bff4af92a050405f68d5d25e910e6ffe1c044b4c239
Oct 30 01:43:50.640079 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:50.640044    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf" event={"ID":"cb2e6fea-4b06-48c2-b00c-2e58f86a2870","Type":"ContainerStarted","Data":"b9f432b65a118608b96c263224f34f72ce96b0c9ecc226c62e7c4effd3d33b94"}
Oct 30 01:43:50.640304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:50.640106    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:50.640304 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:50.640138    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf" event={"ID":"cb2e6fea-4b06-48c2-b00c-2e58f86a2870","Type":"ContainerStarted","Data":"1e0ec63201b27ef640296bff4af92a050405f68d5d25e910e6ffe1c044b4c239"}
Oct 30 01:43:50.688944 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:50.688881    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf" podStartSLOduration=1.275825748 podStartE2EDuration="1.688864991s" totalImagesPullingTime="413.039243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:50.687497652 +0000 UTC m=+3672.235188149" watchObservedRunningTime="2026-10-30 01:43:50.688864991 +0000 UTC m=+3672.236555487"
Oct 30 01:43:56.342821 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:56.342784    5104 scope.go:122] "RemoveContainer" containerID="8747e9b844aeb434604d6f7315befee23222656536f95baf590e6dadc41f0db4"
Oct 30 01:43:56.351898 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:56.351877    5104 scope.go:122] "RemoveContainer" containerID="96a0acc2ae4b1209b95b64e132ac01ca18dc824382f1ac9a487995dc284bdfea"
Oct 30 01:43:56.652498 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:56.652403    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:43:57.732607 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.732567    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-krhjt"]
Oct 30 01:43:57.738584 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.738559    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.769267 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.769234    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-krhjt"]
Oct 30 01:43:57.792773 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.792738    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a815bf7e-a726-4cbd-b10f-e333719640f0-maas-api-tls\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.792990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.792897    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a815bf7e-a726-4cbd-b10f-e333719640f0-openshift-service-ca\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.793062 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.793021    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6j2\" (UniqueName: \"kubernetes.io/projected/a815bf7e-a726-4cbd-b10f-e333719640f0-kube-api-access-cf6j2\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.793120 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.793075    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a815bf7e-a726-4cbd-b10f-e333719640f0-metrics-certs\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.894347 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.894305    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a815bf7e-a726-4cbd-b10f-e333719640f0-openshift-service-ca\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.894574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.894386    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6j2\" (UniqueName: \"kubernetes.io/projected/a815bf7e-a726-4cbd-b10f-e333719640f0-kube-api-access-cf6j2\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.894574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.894438    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a815bf7e-a726-4cbd-b10f-e333719640f0-metrics-certs\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.894574 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.894524    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a815bf7e-a726-4cbd-b10f-e333719640f0-maas-api-tls\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.895158 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.895129    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a815bf7e-a726-4cbd-b10f-e333719640f0-openshift-service-ca\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.897638 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.897609    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a815bf7e-a726-4cbd-b10f-e333719640f0-maas-api-tls\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.897762 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.897677    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a815bf7e-a726-4cbd-b10f-e333719640f0-metrics-certs\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:57.915517 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:57.915485    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6j2\" (UniqueName: \"kubernetes.io/projected/a815bf7e-a726-4cbd-b10f-e333719640f0-kube-api-access-cf6j2\") pod \"maas-api-595cc648c-krhjt\" (UID: \"a815bf7e-a726-4cbd-b10f-e333719640f0\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:58.032354 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:58.032253    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"]
Oct 30 01:43:58.037104 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:58.037067    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zd2vb"]
Oct 30 01:43:58.053065 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:58.053035    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:58.126314 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:58.126277    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:43:58.351422 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:58.351393    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-krhjt"]
Oct 30 01:43:58.353403 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:43:58.353372    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda815bf7e_a726_4cbd_b10f_e333719640f0.slice/crio-b12740654279cfe620af815ba7019890179501f6d3febdb0bf14278e09557200 WatchSource:0}: Error finding container b12740654279cfe620af815ba7019890179501f6d3febdb0bf14278e09557200: Status 404 returned error can't find the container with id b12740654279cfe620af815ba7019890179501f6d3febdb0bf14278e09557200
Oct 30 01:43:58.582252 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:58.581846    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82870ec3-d036-4b87-94a8-12290a4ddc42" path="/var/lib/kubelet/pods/82870ec3-d036-4b87-94a8-12290a4ddc42/volumes"
Oct 30 01:43:58.689312 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:58.689258    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt" event={"ID":"a815bf7e-a726-4cbd-b10f-e333719640f0","Type":"ContainerStarted","Data":"b12740654279cfe620af815ba7019890179501f6d3febdb0bf14278e09557200"}
Oct 30 01:43:59.750481 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:59.750407    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt" event={"ID":"a815bf7e-a726-4cbd-b10f-e333719640f0","Type":"ContainerStarted","Data":"18ad6432c50ab18503b4cbcd9c011a0c9ffb63a6493a7afd1bd4e671d6974c31"}
Oct 30 01:43:59.763509 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:59.761589    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:43:59.814480 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:43:59.804619    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt" podStartSLOduration=2.470557699 podStartE2EDuration="2.80459867s" totalImagesPullingTime="334.040971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:59.803946822 +0000 UTC m=+3681.351637321" watchObservedRunningTime="2026-10-30 01:43:59.80459867 +0000 UTC m=+3681.352289167"
Oct 30 01:44:06.037155 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:06.037115    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:44:06.803737 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:06.803683    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-krhjt"
Oct 30 01:44:06.890512 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:06.890414    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"]
Oct 30 01:44:06.899107 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:06.896688    5104 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf" podUID="cb2e6fea-4b06-48c2-b00c-2e58f86a2870" containerName="maas-api" containerID="cri-o://b9f432b65a118608b96c263224f34f72ce96b0c9ecc226c62e7c4effd3d33b94" gracePeriod=30
Oct 30 01:44:08.950434 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:08.950398    5104 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-92m4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:44:08.951131 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:08.951076    5104 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-92m4m" podUID="02175766-c6b6-45ca-86e2-fc737cc0cfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:44:09.093701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.093666    5104 generic.go:356] "Generic (PLEG): container finished" podID="cb2e6fea-4b06-48c2-b00c-2e58f86a2870" containerID="b9f432b65a118608b96c263224f34f72ce96b0c9ecc226c62e7c4effd3d33b94" exitCode=0
Oct 30 01:44:09.093977 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.093952    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf" event={"ID":"cb2e6fea-4b06-48c2-b00c-2e58f86a2870","Type":"ContainerDied","Data":"b9f432b65a118608b96c263224f34f72ce96b0c9ecc226c62e7c4effd3d33b94"}
Oct 30 01:44:09.139195 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.138373    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:44:09.142637 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:44:09.142564    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:44:09.208061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.207108    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-maas-api-tls\") pod \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") "
Oct 30 01:44:09.208061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.207214    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-openshift-service-ca\") pod \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") "
Oct 30 01:44:09.208061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.207260    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-metrics-certs\") pod \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") "
Oct 30 01:44:09.208061 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.207327    5104 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-kube-api-access-59szn\" (UniqueName: \"kubernetes.io/projected/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-kube-api-access-59szn\") pod \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\" (UID: \"cb2e6fea-4b06-48c2-b00c-2e58f86a2870\") "
Oct 30 01:44:09.209176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.208855    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-openshift-service-ca" pod "cb2e6fea-4b06-48c2-b00c-2e58f86a2870" (UID: "cb2e6fea-4b06-48c2-b00c-2e58f86a2870"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:09.266482 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.266072    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-maas-api-tls" pod "cb2e6fea-4b06-48c2-b00c-2e58f86a2870" (UID: "cb2e6fea-4b06-48c2-b00c-2e58f86a2870"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:09.268040 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.267991    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-kube-api-access-59szn" pod "cb2e6fea-4b06-48c2-b00c-2e58f86a2870" (UID: "cb2e6fea-4b06-48c2-b00c-2e58f86a2870"). InnerVolumeSpecName "kube-api-access-59szn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:09.338059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.332298    5104 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-maas-api-tls\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:44:09.341772 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.341735    5104 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-openshift-service-ca\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:44:09.342005 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.341982    5104 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59szn\" (UniqueName: \"kubernetes.io/projected/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-kube-api-access-59szn\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:44:09.342836 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.342780    5104 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-metrics-certs" pod "cb2e6fea-4b06-48c2-b00c-2e58f86a2870" (UID: "cb2e6fea-4b06-48c2-b00c-2e58f86a2870"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:09.464539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:09.461123    5104 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2e6fea-4b06-48c2-b00c-2e58f86a2870-metrics-certs\") on node \"ip-10-0-130-169.ec2.internal\" DevicePath \"\""
Oct 30 01:44:10.101236 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:10.101204    5104 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"
Oct 30 01:44:10.101718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:10.101203    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf" event={"ID":"cb2e6fea-4b06-48c2-b00c-2e58f86a2870","Type":"ContainerDied","Data":"1e0ec63201b27ef640296bff4af92a050405f68d5d25e910e6ffe1c044b4c239"}
Oct 30 01:44:10.101718 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:10.101341    5104 scope.go:122] "RemoveContainer" containerID="b9f432b65a118608b96c263224f34f72ce96b0c9ecc226c62e7c4effd3d33b94"
Oct 30 01:44:10.151685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:10.151650    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"]
Oct 30 01:44:10.153701 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:10.153673    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-r5dwf"]
Oct 30 01:44:10.576484 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:10.576422    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2e6fea-4b06-48c2-b00c-2e58f86a2870" path="/var/lib/kubelet/pods/cb2e6fea-4b06-48c2-b00c-2e58f86a2870/volumes"
Oct 30 01:44:20.214294 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:20.214246    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:44:39.104169 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:44:39.104129    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:44:48.714802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:48.714756    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-76ntd"]
Oct 30 01:44:56.376848 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:56.376811    5104 scope.go:122] "RemoveContainer" containerID="6ed38de059a2235b78fe187b59cfb45f03cf64b3533a268e0ecb875a2053d06a"
Oct 30 01:44:58.893871 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:58.893837    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d6f85488d-567cx_f4a8486d-468f-404b-b9cc-67d667467964/manager/0.log"
Oct 30 01:44:59.373145 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:59.373108    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-z98l9_6df220f5-b138-4085-b1b3-6f9f6f114abd/manager/0.log"
Oct 30 01:44:59.886333 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:44:59.886297    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59cfcff4f-m8snp_dc4722ec-8353-403a-ab6b-864d3bc1b60d/manager/0.log"
Oct 30 01:45:00.908224 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:00.908191    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-dqmtx_6bb17fa7-7f07-436d-891d-b1af006715c5/manager/0.log"
Oct 30 01:45:00.955009 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:00.954966    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-dqmtx_6bb17fa7-7f07-436d-891d-b1af006715c5/copy-manifests/0.log"
Oct 30 01:45:02.330940 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:02.330898    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-krhjt_a815bf7e-a726-4cbd-b10f-e333719640f0/maas-api/0.log"
Oct 30 01:45:02.879757 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:02.879723    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-52kf6_a3d7c799-8744-4a37-a35b-acae0ebe5a1d/revoke-keys/0.log"
Oct 30 01:45:03.066903 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:03.066874    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-bpfbx_7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6/revoke-keys/0.log"
Oct 30 01:45:03.609990 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:03.609961    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-f9vrr_010ebff9-37da-465f-ad44-49fb0fa2d6cd/revoke-keys/0.log"
Oct 30 01:45:03.779031 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:03.778997    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-46dtl_2bbddb32-a01f-49e1-9c10-68009c76f775/revoke-keys/0.log"
Oct 30 01:45:03.957170 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:03.957141    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e2415be5-zbzs4_01d4a868-89d0-4dec-bf0f-5c49e0275793/revoke-keys/0.log"
Oct 30 01:45:06.215570 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:06.215537    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-6r7jm_8afd8118-e084-42a5-baf0-dddd04b92b7f/manager/0.log"
Oct 30 01:45:06.795506 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:06.795474    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-vrmjk_17087ce5-5b2b-4214-83c7-9f1fa5e5f769/manager/0.log"
Oct 30 01:45:07.011510 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:07.011452    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-76ntd_fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e/limitador/0.log"
Oct 30 01:45:08.144685 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:08.144649    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6bdqg_56d942cc-a33a-4bf9-b0f8-baa86d240b19/discovery/0.log"
Oct 30 01:45:08.349123 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:08.349092    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cfd9d8567-cqfzb_0fa71a9d-efce-4f19-addd-c6a2c38b5988/kube-auth-proxy/0.log"
Oct 30 01:45:08.901433 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:08.901397    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-92m4m_02175766-c6b6-45ca-86e2-fc737cc0cfc3/istio-proxy/0.log"
Oct 30 01:45:09.103590 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:45:09.103556    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:45:09.670795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:09.670762    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-fwnrr_042bc9a4-74e8-4187-ac04-6afbf483756d/istio-proxy/0.log"
Oct 30 01:45:09.864542 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:09.864503    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-j726m_cee3fb42-b55e-4887-b109-efbe113cef93/istio-proxy/0.log"
Oct 30 01:45:10.062112 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:10.062031    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-pc85x_9a10bb9b-ade0-4b26-9b51-8133ffad51ff/istio-proxy/0.log"
Oct 30 01:45:10.601059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:10.601021    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-txfcv_9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae/payload-pre-processing/0.log"
Oct 30 01:45:10.960784 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:10.960752    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75647bc475-s8spf_44aeb453-223b-4cfa-a838-6798b6b5aa1c/router/0.log"
Oct 30 01:45:11.677059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:11.677023    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz_955ad4ba-5e31-4758-8666-744d4549da0b/main/0.log"
Oct 30 01:45:11.860649 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:11.860621    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-z2ktb_626e0a50-e9f7-4b0f-aa09-226cba9a0dfa/main/0.log"
Oct 30 01:45:12.396042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:12.396010    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9_96d0232a-84d5-44ff-b2b6-d45f390c392d/main/0.log"
Oct 30 01:45:22.412147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:22.412107    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d6f85488d-567cx_f4a8486d-468f-404b-b9cc-67d667467964/manager/0.log"
Oct 30 01:45:22.837042 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:22.837008    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-z98l9_6df220f5-b138-4085-b1b3-6f9f6f114abd/manager/0.log"
Oct 30 01:45:23.337032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:23.337001    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59cfcff4f-m8snp_dc4722ec-8353-403a-ab6b-864d3bc1b60d/manager/0.log"
Oct 30 01:45:24.444057 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:24.444028    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-dqmtx_6bb17fa7-7f07-436d-891d-b1af006715c5/copy-manifests/0.log"
Oct 30 01:45:24.482803 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:24.482771    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-dqmtx_6bb17fa7-7f07-436d-891d-b1af006715c5/manager/0.log"
Oct 30 01:45:25.692094 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:25.692061    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-krhjt_a815bf7e-a726-4cbd-b10f-e333719640f0/maas-api/0.log"
Oct 30 01:45:26.049555 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:26.049441    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-52kf6_a3d7c799-8744-4a37-a35b-acae0ebe5a1d/revoke-keys/0.log"
Oct 30 01:45:26.219668 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:26.219637    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-bpfbx_7a9c5c15-a579-4a5a-90fd-250aa4f9f3b6/revoke-keys/0.log"
Oct 30 01:45:26.753067 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:26.753032    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-f9vrr_010ebff9-37da-465f-ad44-49fb0fa2d6cd/revoke-keys/0.log"
Oct 30 01:45:26.934196 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:26.934167    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-46dtl_2bbddb32-a01f-49e1-9c10-68009c76f775/revoke-keys/0.log"
Oct 30 01:45:27.119038 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:27.118959    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e2415be5-zbzs4_01d4a868-89d0-4dec-bf0f-5c49e0275793/revoke-keys/0.log"
Oct 30 01:45:28.736349 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:28.736319    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-6r7jm_8afd8118-e084-42a5-baf0-dddd04b92b7f/manager/0.log"
Oct 30 01:45:29.307849 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:29.307789    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-vrmjk_17087ce5-5b2b-4214-83c7-9f1fa5e5f769/manager/0.log"
Oct 30 01:45:29.539010 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:29.538979    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-76ntd_fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e/limitador/0.log"
Oct 30 01:45:30.460996 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:30.460953    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6bdqg_56d942cc-a33a-4bf9-b0f8-baa86d240b19/discovery/0.log"
Oct 30 01:45:30.661025 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:30.660996    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cfd9d8567-cqfzb_0fa71a9d-efce-4f19-addd-c6a2c38b5988/kube-auth-proxy/0.log"
Oct 30 01:45:31.213634 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:31.213599    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-92m4m_02175766-c6b6-45ca-86e2-fc737cc0cfc3/istio-proxy/0.log"
Oct 30 01:45:31.958960 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:31.958919    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-fwnrr_042bc9a4-74e8-4187-ac04-6afbf483756d/istio-proxy/0.log"
Oct 30 01:45:32.156788 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:32.156751    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-j726m_cee3fb42-b55e-4887-b109-efbe113cef93/istio-proxy/0.log"
Oct 30 01:45:32.341429 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:32.341342    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-pc85x_9a10bb9b-ade0-4b26-9b51-8133ffad51ff/istio-proxy/0.log"
Oct 30 01:45:32.860970 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:32.860937    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-txfcv_9c0ad3ff-bb3d-4c7f-b9aa-f9a89c9b59ae/payload-pre-processing/0.log"
Oct 30 01:45:33.220798 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:33.220763    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75647bc475-s8spf_44aeb453-223b-4cfa-a838-6798b6b5aa1c/router/0.log"
Oct 30 01:45:33.735878 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:33.735847    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zdbdz_955ad4ba-5e31-4758-8666-744d4549da0b/main/0.log"
Oct 30 01:45:33.906830 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:33.906797    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-z2ktb_626e0a50-e9f7-4b0f-aa09-226cba9a0dfa/main/0.log"
Oct 30 01:45:34.454085 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:34.454051    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v9dg9_96d0232a-84d5-44ff-b2b6-d45f390c392d/main/0.log"
Oct 30 01:45:39.102806 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:45:39.102769    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:45:42.025059 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:42.025022    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"]
Oct 30 01:45:42.036032 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:42.036000    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-46dtl"]
Oct 30 01:45:42.575552 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:42.575510    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbddb32-a01f-49e1-9c10-68009c76f775" path="/var/lib/kubelet/pods/2bbddb32-a01f-49e1-9c10-68009c76f775/volumes"
Oct 30 01:45:58.027143 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:58.027104    5104 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"]
Oct 30 01:45:58.038360 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:58.038323    5104 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-f9vrr"]
Oct 30 01:45:58.576200 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:45:58.576164    5104 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010ebff9-37da-465f-ad44-49fb0fa2d6cd" path="/var/lib/kubelet/pods/010ebff9-37da-465f-ad44-49fb0fa2d6cd/volumes"
Oct 30 01:46:06.797442 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:06.797410    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59cfcff4f-m8snp_dc4722ec-8353-403a-ab6b-864d3bc1b60d/manager/0.log"
Oct 30 01:46:07.109742 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:07.109649    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59cfcff4f-m8snp_dc4722ec-8353-403a-ab6b-864d3bc1b60d/manager/0.log"
Oct 30 01:46:09.103287 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:46:09.103247    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:46:39.102349 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:46:39.102308    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:46:50.018253 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:50.018214    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59cfcff4f-m8snp_dc4722ec-8353-403a-ab6b-864d3bc1b60d/manager/0.log"
Oct 30 01:46:50.371055 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:50.370947    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59cfcff4f-m8snp_dc4722ec-8353-403a-ab6b-864d3bc1b60d/manager/0.log"
Oct 30 01:46:55.981389 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:55.981350    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nk4k6_b3bb0053-40b7-49df-9490-192e78a9b720/global-pull-secret-syncer/0.log"
Oct 30 01:46:56.298128 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:56.298052    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rrv72_6a4543fe-837e-4480-b419-974b723e4475/connection-checker/0.log"
Oct 30 01:46:56.403856 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:56.403824    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ss447_2f6bf9b3-3ed3-4f5b-87e0-5d7d3c6fa405/connection-checker/0.log"
Oct 30 01:46:56.410962 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:56.410937    5104 scope.go:122] "RemoveContainer" containerID="da6b3fe14919bd8aab277953842ce73bce685d5bc2cd611101c2a69f0197ae57"
Oct 30 01:46:56.420316 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:56.420294    5104 scope.go:122] "RemoveContainer" containerID="d793f31f7a5f706cecd2adf52c7870e4fd5f559b5bfffe083b062392ba1dc56a"
Oct 30 01:46:56.510854 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:56.510827    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xf7xr_79ef76c0-8e88-4b4a-8d6a-83da18b39d78/connection-checker/0.log"
Oct 30 01:46:56.771873 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:56.771837    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-skbfn_7469affe-9d00-44b3-a833-4ca25b86da5c/konnectivity-agent/0.log"
Oct 30 01:46:57.037966 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:46:57.037871    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-169.ec2.internal_c2831292250175cd4fa5c7c5f1d89a07/haproxy/0.log"
Oct 30 01:47:03.853993 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:03.853956    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-6r7jm_8afd8118-e084-42a5-baf0-dddd04b92b7f/manager/0.log"
Oct 30 01:47:04.244614 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:04.244578    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-vrmjk_17087ce5-5b2b-4214-83c7-9f1fa5e5f769/manager/0.log"
Oct 30 01:47:04.343100 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:04.343069    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-76ntd_fee88f31-2ff9-4d35-ae21-9fce7cbcdd8e/limitador/0.log"
Oct 30 01:47:05.563515 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.563476    5104 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"]
Oct 30 01:47:05.564230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.564203    5104 cpu_manager.go:414] "containerMap: removing container" podUID="cb2e6fea-4b06-48c2-b00c-2e58f86a2870" containerName="maas-api"
Oct 30 01:47:05.564230 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.564224    5104 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2e6fea-4b06-48c2-b00c-2e58f86a2870" containerName="maas-api"
Oct 30 01:47:05.564361 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.564322    5104 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2e6fea-4b06-48c2-b00c-2e58f86a2870" containerName="maas-api"
Oct 30 01:47:05.569058 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.569039    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.585696 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.585673    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvzh4\"/\"kube-root-ca.crt\""
Oct 30 01:47:05.585822 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.585696    5104 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvzh4\"/\"openshift-service-ca.crt\""
Oct 30 01:47:05.600675 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.600639    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"]
Oct 30 01:47:05.680816 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.680785    5104 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tvzh4\"/\"default-dockercfg-lc9qq\""
Oct 30 01:47:05.682697 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.682675    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtw2k\" (UniqueName: \"kubernetes.io/projected/65dd1333-10c4-4b8d-8023-92136904a14d-kube-api-access-dtw2k\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.682825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.682719    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-sys\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.682825 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.682781    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-podres\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.682908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.682866    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-lib-modules\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.682908 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.682883    5104 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-proc\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.783795 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.783758    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-podres\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.783984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.783867    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-lib-modules\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.783984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.783897    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-proc\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.783984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.783900    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-podres\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.783984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.783962    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-proc\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.783984 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.783974    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtw2k\" (UniqueName: \"kubernetes.io/projected/65dd1333-10c4-4b8d-8023-92136904a14d-kube-api-access-dtw2k\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.784176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.784003    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-lib-modules\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.784176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.784028    5104 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-sys\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.784176 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.784059    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65dd1333-10c4-4b8d-8023-92136904a14d-sys\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.802377 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.802341    5104 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtw2k\" (UniqueName: \"kubernetes.io/projected/65dd1333-10c4-4b8d-8023-92136904a14d-kube-api-access-dtw2k\") pod \"perf-node-gather-daemonset-zr8lp\" (UID: \"65dd1333-10c4-4b8d-8023-92136904a14d\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:05.881771 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:05.881688    5104 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:06.175541 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:06.175348    5104 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"]
Oct 30 01:47:06.178044 ip-10-0-130-169 kubenswrapper[5104]: W1030 01:47:06.178017    5104 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod65dd1333_10c4_4b8d_8023_92136904a14d.slice/crio-cf25283009881699eb96209dee21776960efaafb8265a641e0358ef15c51c86e WatchSource:0}: Error finding container cf25283009881699eb96209dee21776960efaafb8265a641e0358ef15c51c86e: Status 404 returned error can't find the container with id cf25283009881699eb96209dee21776960efaafb8265a641e0358ef15c51c86e
Oct 30 01:47:06.628265 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:06.628180    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-vfg9g_2deae516-c4e0-4377-98ee-084581c4b027/cluster-monitoring-operator/0.log"
Oct 30 01:47:07.097923 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:07.097880    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp" event={"ID":"65dd1333-10c4-4b8d-8023-92136904a14d","Type":"ContainerStarted","Data":"303de863533cb745029feb2883ca82353cd3b52a5af901d68c1d755e10284c6b"}
Oct 30 01:47:07.098147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:07.097952    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:07.098147 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:07.097976    5104 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp" event={"ID":"65dd1333-10c4-4b8d-8023-92136904a14d","Type":"ContainerStarted","Data":"cf25283009881699eb96209dee21776960efaafb8265a641e0358ef15c51c86e"}
Oct 30 01:47:07.135755 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:07.135700    5104 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp" podStartSLOduration=2.135682407 podStartE2EDuration="2.135682407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:07.134438798 +0000 UTC m=+3868.682129295" watchObservedRunningTime="2026-10-30 01:47:07.135682407 +0000 UTC m=+3868.683372881"
Oct 30 01:47:07.281616 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:07.281580    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4lnp9_276c6084-0984-48d8-886a-9ef52e789339/node-exporter/0.log"
Oct 30 01:47:07.389337 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:07.389250    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4lnp9_276c6084-0984-48d8-886a-9ef52e789339/kube-rbac-proxy/0.log"
Oct 30 01:47:07.503802 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:07.503769    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4lnp9_276c6084-0984-48d8-886a-9ef52e789339/init-textfile/0.log"
Oct 30 01:47:09.104291 ip-10-0-130-169 kubenswrapper[5104]: E1030 01:47:09.104254    5104 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lmwj7" containerName="extractor"
Oct 30 01:47:09.601834 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:09.601804    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-wfw4k_7a355cec-6892-4b36-89b7-60c6b3460bf0/prometheus-operator-admission-webhook/0.log"
Oct 30 01:47:11.926829 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:11.926793    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-s248j_85585df9-d495-4d44-bd82-398d44b76cf8/networking-console-plugin/0.log"
Oct 30 01:47:12.925766 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:12.925732    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/2.log"
Oct 30 01:47:12.933862 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:12.933837    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5phkz_b9dac87d-7a94-43b5-ae69-38f6975388ed/console-operator/3.log"
Oct 30 01:47:13.114382 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:13.114351    5104 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-zr8lp"
Oct 30 01:47:13.954002 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:13.953968    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-9ks9h_d581f7f9-2281-4635-a1ef-4930a5ad7026/download-server/0.log"
Oct 30 01:47:15.075808 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:15.075772    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-56c4z_9cec9dfe-c016-4228-be90-4724120f7260/volume-data-source-validator/0.log"
Oct 30 01:47:17.660135 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:17.660104    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dkpn9_49fcf677-f503-492a-90e8-c95e329ee6f6/dns/0.log"
Oct 30 01:47:17.765468 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:17.765435    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dkpn9_49fcf677-f503-492a-90e8-c95e329ee6f6/kube-rbac-proxy/0.log"
Oct 30 01:47:18.098739 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:18.098707    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rxj86_7d027727-5c89-44a2-b1b8-20661fd13909/dns-node-resolver/0.log"
Oct 30 01:47:19.185853 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:19.185817    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7bbf7cb548-clh8s_55c2f131-88b8-4125-b2c9-be2d4704c95b/registry/0.log"
Oct 30 01:47:19.507001 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:19.506913    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r6gln_e1c4505b-9dd7-4044-8c27-b1f6e2d59ea6/node-ca/0.log"
Oct 30 01:47:24.531539 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:24.531497    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6bdqg_56d942cc-a33a-4bf9-b0f8-baa86d240b19/discovery/0.log"
Oct 30 01:47:24.569046 ip-10-0-130-169 kubenswrapper[5104]: I1030 01:47:24.569008    5104 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cfd9d8567-cqfzb_0fa71a9d-efce-4f19-addd-c6a2c38b5988/kube-auth-proxy/0.log"